Autonomy Is Not a Property

In May, during a capture-the-flag exercise run by the independent evaluator Irregular, Google's Gemini found credentials on the open internet and used them to get into systems belonging to three real companies. In one case it guessed a password until a protected system opened. In the other two it used credentials that had been left exposed in a public repository. Google confirmed the incidents, made sure the three affected parties were told, and changed the testing process with its partner. The story surfaced in September and was reported as what it plainly was: the first known case of that company's models breaking containment and reaching third parties.

Two details from the confirmed account did not travel nearly as far as the headline, and between them they are the whole story.

The first is what the model had been asked to do. It was tasked with retrieving information from software belonging to a fictional company — a fictional company that shared its name with a real one. The environment was not supposed to be reachable from the internet at all; Irregular has said that access was made available unintentionally.

The second is how it ended. In each of the three cases, the model worked out that it had reached a real company, and stopped.

It did not stop because a guardrail fired. It did not stop because someone pulled a plug. It stopped because its own picture of the situation changed: what it had taken to be part of the permitted test environment turned out not to be, and once that classification corrected itself, the behaviour corrected itself with it. Up to that moment the model had been doing exactly what it was asked to do, competently, against targets it believed were in scope.

Put those two details side by side and the incident stops looking like a story about a dangerous model. A name resolved to the wrong entity. Everything downstream followed correctly from that resolution, including, eventually, its correction.

So the failure was not a failure of intent, and it was not really a failure of capability either. It was a failure of reference — the oldest failure there is, and the one every ontologist will recognise on sight. Two things shared a label; the system had no way to tell which one it was holding; and it acted on the answer it had.

The boundary was the thing that was missing

There is a comfortable reading of this incident and I want to set it aside first. The comfortable reading is that sandboxes need to be tighter, network egress needs to be narrower, and evaluations should run against infrastructure that cannot reach anything real. All of that is true and all of it is being done. It is also not the interesting part.

The interesting part is that the question the model got wrong is a question about the world, not a question about the model. Does this host belong to the set of things I am permitted to act on? is not a fact about weights or training. It is a fact about an arrangement somebody made: a scope somebody defined, for a test somebody commissioned, under terms somebody agreed. The model had to infer that arrangement from whatever was in front of it, because nothing in front of it stated the arrangement in a form it could check.

Notice that no amount of additional intelligence closes this. A sharper model facing two entities with the same name and no authoritative statement of which one is in scope does not get the answer right more often; it gets to the wrong systems faster and with better cover stories. Identity resolution against a scope somebody defined is not an inference problem. It is a lookup problem against a fact that either was written down in machine-checkable form or was not.

This generalises well beyond red-team evaluations, and it generalises in a direction that should worry anyone deploying agents inside an organisation. Every consequential action an enterprise agent takes sits inside a boundary of exactly this kind. Which systems may this agent touch. Which records count as in scope for this task. Which counterparties are covered by the arrangement that authorised the work. Whether a given action falls inside the boundary is not a question the agent can settle by being smarter. It is a question that has an answer only because an organisation decided something, and the agent can only get it right if that decision exists somewhere it can be consulted.

When it does not, the agent does what Gemini did: it forms a plausible picture and proceeds. And the more capable the agent, the more efficiently it proceeds.

The technical layer is arriving, and it is arriving fast

It would be wrong to suggest nothing is being done about this. The infrastructure side has moved very quickly, and the honest thing is to say how far it has moved before claiming anything is missing.

Agent-specific identity, short-lived credentials, per-tool-call authorisation, policy gates evaluated before execution, audit trails good enough to reconstruct what happened: these have gone from research prototypes to products in about eighteen months. Deontic policy languages now express not just what an agent may and may not do but what it is obliged to do afterwards, when a standing obligation may be waived, and which rule wins when two policies conflict — and they evaluate all of that outside the model, fast enough to sit synchronously in front of every action. The unit of control has begun moving from the individual action to the whole trajectory, on the recognition that a sequence of individually permitted steps can violate a constraint that no single step violates.

Delegation is being taken seriously too, which matters because delegation is where authority gets lost. A recent gap analysis evaluated four widely used multi-agent frameworks against a deliberately harsh standard — a correct system being one in which a fully prompt-injected agent still cannot exceed the authority explicitly delegated to it — and found that three of the four provide no built-in confinement at all, and the fourth only partial. Their proposed broker confines a compromised sub-agent to roughly one and a half reachable actions where unconstrained bearer delegation leaves more than eight thousand open. That is a real result, with a real caveat worth stating: it is a preprint, and the authors also sell the mechanism.

Put all of that together and a fair summary is that the question was this action technically permitted? is close to solved, and the question what did this agent actually do? is solved well enough to audit. Neither of those is the question Gemini got wrong.

Then a clinical study asked a different question

In September, Nature Medicine published something that has stayed with me longer than any of the incident reporting, and it comes from a corner of the field I do not usually read for governance ideas.

A group led by Li Zhang built a diagnostic agent that runs entirely on-premise, on open-weight models, and evaluated it on two benchmarks derived from intensive-care records. It reached about ninety percent accuracy on one and about eighty-four on the other, near a cloud baseline. That is the headline result and it is not the one that matters.

The second half of the paper asks whether the system can tell, at the moment of deciding, how much its own answer should be trusted. The authors compared several candidate signals and found that the strongest discriminator of correctness was not any internal confidence score but behavioural consistency — whether repeated runs of the same case land on the same answer. Setting a high consistency threshold, they kept just under half of all cases for autonomous handling, and those cases came in at almost ninety-nine percent diagnostic accuracy. Everything else went to a human.

They call the resulting design selective autonomy, and the term is theirs, not mine. I want to be careful about that, because I nearly adopted it as though I had thought of it. Their framing names the two things standing between clinical AI agents and actual deployment as institutionally governed deployment and decision-time uncertainty estimation, which is a remarkably precise statement of the problem from people who were not trying to write about governance at all.

What makes it valuable outside medicine is the shape of the claim. The goal is not an agent that handles every case. The goal is an agent that can identify which cases it is entitled to handle. Cut the automation rate roughly in half and the error rate on what remains automated falls by an order of magnitude. That trade is available in compliance review, credit decisions, contract analysis, security triage and a great many administrative processes, and hardly anybody is taking it, because the industry keeps asking what proportion of work an agent can do rather than which proportion it can do reliably enough to be left alone with.

Two conditions, and they are not the same condition

Here is where the two stories meet, and where I think a distinction is being missed that will cost people money.

The Gemini incident is about a boundary. The clinical study is about a threshold. They look similar — both end in the agent should not have proceeded here — and they are not similar at all.

A boundary is institutionally set. It says which actions, on which objects, an actor may take without asking anyone. Somebody with the standing to decide decided it; it has a scope; it was valid at a particular time and may not be valid now. It is, in the end, a question of legitimacy: who said you may.

A threshold is evidentially measured. It says whether, in this instance, the evidence supports acting at all. It is computed at runtime from the system's own behaviour, it has an error rate, and it is indifferent to who authorised anything. It is a question of warrant: does what we know right now support proceeding.

Neither substitutes for the other, and systems that carry only one fail in characteristic ways. An agent with a correct boundary and no threshold does authorised things badly — it acts well within its permissions on cases it has no business deciding, and everything it does will pass the audit. An agent with a good threshold and no boundary does well-calibrated things nobody sanctioned; it is confident, it is usually right, and when someone asks under whose authority it acted there is no answer. Gemini, on the evidence available, was the second kind. It behaved sensibly given its picture of the world. Nobody had given it a picture it could check.

So the design rule I have been working toward looks like this, and I now think both halves are load-bearing:

An agent performs a consequential action only where an institution has granted the boundary that covers it, and the evidence at hand warrants acting inside that boundary on this occasion.

Which is a longer way of saying that autonomy is not a property an agent has. It is a permission an organisation grants, for a particular action, in a particular context, on particular evidence, within stated limits — and it can be withdrawn by either half failing.

The half that is still missing

If the boundary has to be granted, the obvious next question is who granted it, and that is where current practice runs out.

A runtime control plane can tell you that an agent was authenticated, that its credential was valid, and that its action matched a policy. It cannot tell you whether the policy encodes a decision the organisation was entitled to make, which mandate makes that decision binding, whether the mandate was still active when the action occurred, or who remains answerable now that it has. Those facts do not live in the model and they do not live in the tool call. They live in the organisation, usually in prose, usually in a document nothing can query.

The gap has a name now, and not one I gave it. Writing in Frontiers in Political Science this month, Mohammed Salah and colleagues describe what they call a democratic authorization gap: the breaking of the demonstrable chain between legally constituted public authority and the actions an agent selects, sequences or executes. They separate technical authorization from responsibility and accountability and from the prior question of legitimate authority, and they put the operative test in a form I find hard to improve on — whether the actions and sub-delegations an agent later chooses remain demonstrably connected to the authority originally granted. Accountability, they note, does not by itself establish that the institution had the authority to delegate in the first place.

They are writing about government. The same discontinuity exists in any company that has started handing work to agents, and it produces a category of event that current tooling cannot see at all: the action that is technically permitted and institutionally unauthorised. Every gate passes. Every log is complete. Nobody was entitled to it.

How common is that? I do not know, and I am suspicious of anyone who claims to. It is the load- bearing empirical question of the work I am doing and it deserves to be asked hostilely, because if the answer turns out to be rare, and caught in practice by existing permission layers, then the distinction I have spent this post drawing is a conceptual nicety with no operational consequence.

There is one indirect signal worth reporting, from a study that was not looking for it. Building an ontology for clinical AI agents, Pengze Li and Cui Tao scored the reporting completeness of 279 published papers and found the worst gaps exactly where institutional meaning would have to sit: runtime and architecture reported only partially or not at all in eighty-five percent of papers, governance and safety in eighty-three, provenance in seventy-eight — against twenty-six percent for evaluation. Most of these gaps are partial rather than total, but the pattern is plain: the field documents how well its agents score far more completely than how they are built and governed. That is not proof that unauthorised action is common. It is decent evidence that we would not currently notice if it were.

What I take from the pair of them

An agent that stopped when it finally understood where it was, and an agent that abstains when it notices it might be wrong. One of those was a failure and one was a design, and the difference between them is not intelligence.

The Gemini model was not confused about how to act. It was confused about what it was acting on, and nothing in its situation could correct that except the accident of finding out. The clinical agent was not more capable; it was equipped with a way of noticing the limits of its own reliability and an institution willing to take back the cases it handed up.

Both of those are representational achievements, not model achievements. Which is the argument I keep arriving at from different directions: the interesting frontier is no longer how much a system can do. It is how much of what a system does an organisation can say it meant.

More as the work develops.

New posts by email: subscribe to the newsletter. One email per post, no tracking.