What AI Coding Agents Break When Nobody Is Looking

Your coding agent fixed the bug. It also changed four things you never asked it to touch, and your tests passed anyway.

We measured how often that happens across 65 models from 22 labs, on 1,504 trials. Then we measured what happens when the agent is handed a behavioral diff of what its last change actually did.

Hallucinations fell on all 64 of the models we could score. On the typical model they fell 77.3%. Across the whole corpus, pooled over every trial, they fell 75.3%.

What we mean by a hallucination

Unintended changes, additions or deletions to working code the agent was never asked to touch.

That is a narrower definition than the industry usually uses, and it is deliberate. We are not counting wrong facts or invented APIs. We are counting the case where you asked for one thing, the agent did that thing, and it also quietly altered behavior somewhere else. The metric is introduced_new_path: a request that returned one result before the change and a different one after, on an endpoint nobody was working on.

That is the failure that survives code review, because the diff looks reasonable and the tests are green.

How the measurement works

Every model runs the same task twice. Once unaided. Once with a behavioral diff of the previous change in its prompt. Same benchmark, same bugs, same harness.

Afterwards we replay real traffic against the patched service and compare it to the unmodified baseline, request by request. Any endpoint that answers differently and was not part of the assignment counts as introduced.

Each change is traced back to the edit that caused it: we remove one edit at a time and replay, so one broken helper touched by forty endpoints is one problem, not forty. Raw per-endpoint counts inflate unevenly enough to reorder which labs look worst, so we do not use them.

It does not work equally everywhere

No model got worse with ReGrade in the loop, and none stayed flat. Some barely moved, though. Ling 2.6-1T fell 2.8%, MiniMax M2.5 fell 5.6% and Haiku 4.5 (Aug 2026) fell 5.9%. We name them because a claim that holds on 64 of 64 is worth more when you can see where it holds weakest.

An earlier cut of this corpus, at 63 models and 1,451 trials, reported eight exceptions and named seven of them. A later cut, at 69 models and 1,577 trials, reported one: DeepSeek V4-pro-0813, which was not among those seven. That model now falls 38.5%, and we know why. Four of its trials had been re-run after they were scored, and the published figure still used the old scores. Scored on the patches the model actually wrote, it improves. The seven earlier exceptions we have not traced one by one. Between cuts, thin models were deepened from three trials an arm to ten, several models shipped new versions, and trials that escaped the test sandbox were excluded. Any of those could account for them, and we would rather leave that open than pick the flattering explanation.

One more model is unscored. gpt-oss-20b fixed nothing when working unaided and broke nothing, so there is no baseline to fall from. With ReGrade it fixed 3.8 bugs a run on average, at a cost of 0.4 introduced changes.

Where it stops completely

On 16 models across 9 labs, introduced changes went to zero. Not lower. Zero.

That list includes current flagships: GPT-5.5, GPT-5.6 Sol, Opus 5, Opus 4.8, Fable 5, Grok 4.5, Gemini 3.1 Pro, Qwen3.7-Max and Kimi K2.7-code. It also includes Nvidia's nemotron-3.5-lightning and Sakana's fugu-ultra.

What to take from this

Capability does not solve it. The strongest models in the world introduce unintended changes when they work blind, and stop when they can see what their last change did.

If you are running coding agents in anger, the question is not which model breaks least. It is whether your agent can see its own behavioral diff before it writes the next patch.

Every number here is reproducible. The card that carries these claims refuses to build if the corpus stops supporting them.