I met a comedian at a wedding the other weekend where I knew almost no one and was busy making friends. He offered to be my third, and before long he was telling me, with some pride, how he was getting on in his career. Not famous, he said, but the kind of comedian others admire. We got onto AI somehow, as one does now, and according to him, most of the comics he knew were writing their jokes with it, and the jokes were good. Not him, of course. Well, not for the rooms he cared about. For the corporate work, some Swiss bank handing out its annual awards to itself, he might let a model take the first pass, because who is really listening. But not for the real work. He was very clear about that, in the way people are always very clear about the thing they have just admitted they sometimes do.

He at least had a rule, and I don’t seem to have one at all. This is a subject I care about, written for people I would quite like to think well of me, and I wrote it with AI. Almost every word of it. Can you tell?

I’m not certain I can, and it’s mine. I used to be able to spot the tells, or thought I could, and I am less sure of that every month. It comes back clean and about right, often enough that I keep it, and nothing in it says what made it.

So why do we keep trying to make agents announce themselves?

We ask them to tick a box confirming they are bots, on the cheerful assumption that the ones worth worrying about would feel any obligation to be honest. We have tried versions of this for years, most visibly with the CAPTCHA, the small daily ritual in which we click on fire hydrants and blurry shopfronts to prove we are human. Machines now pass it more reliably than we do, somewhere between 96 and 100 per cent of the time against our own 50 to 85. We invented the test, and we have ended up worse at it than the things it was built to keep out.

And the incentives all run the other way. An agent that declares itself honestly gets throttled, or blocked, or quietly charged more for the privilege, so anything with a reason to lie will lie, and the only ones left wearing the name tag are the ones who were never going to give you any trouble. I don’t think I’m a bad actor, and I don’t follow the rules either. I assured more than one website that I was over eighteen at a moment when that was not strictly true, and so did you. An agent does the same, only faster. Regulators have noticed, and the EU, California and China are starting to require disclosure, which will work beautifully on the agents that were already going to behave.

We have to build better than we used to. For as long as you could assume a competent person on the other end you could, occasionally, be a little sloppy and get away with it. Your friendly human user mostly covered for you: they read the ambiguous affordance the way you meant it, worked around the confusing default, never chose the option that was obviously a mistake. That judgement sat in the user, and it quietly subsidised every gap you left in the product. You could let one wrong click cost a popular project a decade of community, and trust that maintainers mostly wouldn’t make it by accident. In return, you gave them a pretty picture of a horse on a bicycle when they got through all their notifications, because they liked that.

An agent doesn’t subsidise you. It does exactly what the interface tells it, at speed and at scale, with the literal-minded enthusiasm of a new hire who has read the manual and nothing else, so the affordance you left ambiguous gets taken at its word ten thousand times over and the bad default quietly becomes the house style. You can see it in the code these things write. Someone tested a large batch of it recently and close to half carried a known security hole, the sort a careful reviewer notices on the way past to something else. The newer models have not brought that down. They have got better at sounding right, which is a different and more dangerous skill, so the code now arrives fluent and confident and tidy and wrong at roughly the same rate as before, reading like the work of someone who has never once been unsure. The judgement that used to catch this lived in the person reading the change before it shipped, and that person is increasingly not in the room. Shutting the agents out was never on offer. What is left is to put the judgement into the software on purpose, because nothing downstream is going to supply it for you.

None of this stays politely inside the API, either. The comforting division of labour, where agents go round the back to the machine entrance and humans come through the front in their Sunday clothes, holds right up until your beautifully considered screen turns out to be the quickest path to the thing an agent wants, at which point it will use the screen the same as it would use anything else, and your front door becomes one more endpoint that happens to have buttons on it.

The user on the other end is, more and more often, not a person. This isn’t a forecast: somewhere in 2024, by most measures, more than half of all web traffic stopped being human, and the line has kept climbing since. The person is often still there, further back, sending a model ahead to do the reading and the first pass, then trusting whatever comes back.

The people who keep open-source projects alive, the maintainers, mostly a handful of volunteers, have been asking for a tool that would tell them which of the strangers filing bug reports and proposing changes to their projects are actually human. They are not wrong to want it, and a filter would be the obvious relief from the flood they are buried under, where by one maintainer’s tally only one AI pull request in ten was worth keeping. But no such tool can be built, because the actor behind a request is not observable and every incentive works to keep it that way, so what they are asking for was always going to solve far less than it promised, and the disappointment is real.

I wanted the same certainty, and I held onto it longer than I should have. Not long ago I sat through a review of how we handle pull requests, careful work, full of honest argument about how much of a diff a reviewer really reads before they start skimming, and how to keep the changes that matter in front of them. Most of its assumptions were already wrong, and so were mine, because a growing share of what opens those pull requests, and what reviews them, was never going to read them like a person at all. I had assumed, without ever deciding to, that the human was the default and the agent was the thing we would deal with later. I had it the wrong way round. Building for people was never where we struggled. What we are slow to notice is the day the people stop being the ones who turn up.

The bill for that is already arriving, and you can read it in the security holes and in the maintainers closing their doors. The answer is not a mystery either. You put the judgement back into the software as the things it will not do: the confirmation that knows the difference between an empty repository and a decade of community, a permission scoped so it cannot reach further than it should, an action you can take back. There are guardrail frameworks for exactly this now, the kind that check an agent’s actions before they run, and by one recent survey only about one in five companies has much of it in place, while nearly three in four prepare to hand real work to agents anyway. They are still building for someone who has already left the room.