AI coding hallucinations
Ongoing testingIn our ongoing testing, ReGrade® reduces AI coding hallucinations by over 83%.
ReGrade records real traffic, replays it against your candidate version and reports what changed, for engineering leaders who approve releases.
An AI coding hallucination is an unintended change, addition or deletion to working code that the agent was never asked to modify.
Curtail research · 84 models, 22 labs, updated 2026-10-24
Every AI lab we tested produced AI coding hallucinations. All 22 labs. The strongest models included.
With ReGrade AI coding hallucinations dropped to zero for these models.
A hallucination is counted by replaying each trial's saved patch offline and counting the changes that fall outside the bugs the model was asked to fix. Each change is traced back to the edit that caused it, by removing one edit at a time and replaying, so one mistake that surfaces at nine call sites counts once. On the typical model, ReGrade's patented behavioral comparison cut those changes by over 83%. In the main comparisons, no unintended changes were observed in the ReGrade trials on the 28 models below, across 11 labs. Zero observed changes is not a guarantee on future code.
- OpenAIGPT-6 Astra, gpt-6.1-sol, GPT-6 Sol, GPT-5.5, GPT-5.6 Sol, gpt-5.6-terra
- XiaomiMiMo V2.6 Pro
- xAIGrok 4.5, Grok 3 Fast
- Metamuse-wonder, Muse Spark 1.3, muse-spark-open
- Sakanafugu-ultra-v2, fugu-ultra
- AnthropicFable 5, Opus 5.5, claude-haiku-5-5, Fable 5.1, Opus 4.8, Opus 5, Opus 4.6
- Googlegemini-4-argon, Gemini 3.1 Pro Preview
- MoonshotKimi K2.7 Code
- DeepSeekdeepseek-v4.1-flash-1017
- AlibabaQwen3.8 Max Prime, Qwen3.7 Max
- NvidiaNemotron 3.5 Lightning
Every model tested, on AI coding hallucinations
| Hallucinations | Bug Fixes | |||||||
|---|---|---|---|---|---|---|---|---|
| Verdict | ||||||||
| gpt-6-astra | 6.33 | 0 | −100% | 14.7 | 18.0 | OpenAI | 3 | ✓ |
| gpt-6-sol | 3.00 | 0 | −100% | 14.3 | 18.0 | OpenAI | 3 | ✓ |
| gpt-6.1-sol | 5.00 | 0 | −100% | 14.3 | 18.0 | OpenAI | 3 | ✓ |
| muse-wonder | 3.67 | 0 | −100% | 13.3 | 17.3 | Meta | 3 | ✓ |
| minimax-m3 | 1.33 | 0.33 | −75% | 7.3 | 17.7 | MiniMax | 3 | ✓ |
| opus-4.6 | 1.33 | 0 | −100% | 7.0 | 17.7 | Anthropic | 3 | ✓ |
| longcat-2.0 | 3.00 | 0.33 | −89% | 6.7 | 11.0 | Meituan | 3 | ✓ |
| glm-5.2 | 1.85 | 0.74 | −60% | 6.7 | 16.6 | Z.ai | 19 | ✓ |
| mistral-large-4 | 1.33 | 1.33 | 0% | 5.3 | 14.7 | Mistral | 3 | no change |
| laguna-xs-2.1 | 1.80 | 1.60 | −11% | 1.8 | 10.3 | Poolside | 15 | ✓ |
| ling-2.6-1t | 1.80 | 1.75 | −3% | 1.6 | 7.0 | InclusionAI | 4 | ✓ |
| nemotron-3.5-lightning | 1.25 | 0 | −100% | 1.4 | 5.3 | Nvidia | 3 | ✓ |
| grok-3-mini | 0.67 | 0.33 | −50% | 1.0 | 2.8 | xAI | 3 | ✓ |
✓ improved · * got worse · N/A not scorable. Click a column to sort. Scroll sideways to see the lab and sample-count columns.
About this table
N/A: the model did not edit the file in the control run, so there is no baseline to compare against. The table lists the models with a measurement on this metric, from a corpus of 84 models, 22 labs and 1,633 agent-mode trials. The provider-pinned comparisons add 30 separately labeled trials, for 1,663 total. Min. replays/arm is the smaller eligible replay count of the control and treatment arms; individual arms can contain more runs. Bug fixes are unaided (mean bugs fixed across the model's control-arm trials) and aided (the same average over the treatment trials), so the table can be sorted by how capable a model is on its own and by how far it gets with help. Both are our own measurements. Repair and damage eligible subsets can differ, so bug-fix averages need not use every replay counted here. A dash means the model sits below the trial floor, so no figure is published for it.
The honest caveats. On 1 model the count did not move: mistral-large-4. The main comparison results are in the table above, sortable and searchable. The provider-pinned results follow separately, with three trials per arm: July DeepSeek on Baidu reached zero observed changes and Nemotron Ultra's count rose. Those samples do not establish stable effects.
Provider-pinned comparisons
Five separate cells, 30 trials, three trials per arm. Both arms of each cell used the same provider. These batches remain separate from the historical model comparisons above and their headline statistics. Every arrow reads without ReGrade → with ReGrade.
| Model | Provider and precision | Bugs fixed / 18 | Causal damage per trial |
|---|---|---|---|
| deepseek-v4-flash-0423 | DeepInfra · fp8 | 4.33 → 16.00 | 1.33 → 0.33 |
| deepseek-v4-flash-0731 | Baidu · fp8 | 5.67 → 17.00 | 2.67 to 3.33 → 0.00 |
| nemotron-3-ultra | BaseTen · fp4 | 4.33 → 15.00 | 1.33 → 1.67 |
| inkling | Together · precision unknown | 8.67 → 14.33 | 3.67 → 1.67 |
| glm-5.2 | Baidu · fp4 | 7.00 → 16.67 | 2.00 to 3.00 → 1.00 |
Higher repair scores and lower damage are better. Ranges include unresolved hunk-removal variants. Nemotron introduced slightly more damage with ReGrade. Three trials per arm do not establish stable model effects.
Works with every major coding agent and model API.
5 coding-agent CLIs · 84 models · 22 labs.
Run through Claude Code, Codex CLI, qwen-code, Kimi Code CLI, Mistral Vibe CLI.
Coding-agent CLIs5 CLIs · 27 models
Claude Code
- claude-haiku-5-5
- fable-5
- fable-5.1
- glm-5.1
- haiku-4.5
- haiku-4.5 (Aug 2026)
- opus-4.6
- opus-4.7
- opus-4.8
- opus-5
- opus-5.5
- sonnet-4.6
- sonnet-5
- sonnet-5.5
Codex CLI
- gpt-5.4
- gpt-5.5
- gpt-5.6-luna
- gpt-5.6-sol
- gpt-5.6-terra
- gpt-6-astra
- gpt-6-luna
- gpt-6-sol
- gpt-6.1-sol
qwen-code
- qwen3-coder
- qwen3-coder-plus
Kimi Code CLI
- kimi-k26
Mistral Vibe CLI
- devstral-2
Models tested via API, by lab57 models · 21 labs
xAI
- grok-3-fast
- grok-3-mini
- grok-4-1-fast-non-reasoning
- grok-4.20-0309-reasoning
- grok-4.3
- grok-4.5
- grok-4.7
- grok-build-0.1
DeepSeek
- deepseek-v4-flash-0423
- deepseek-v4-flash-0731
- deepseek-v4-pro (Sep 2026)
- deepseek-v4-pro-0423
- deepseek-v4-pro-0813
- deepseek-v4.1-flash
- deepseek-v4.1-flash-1017
Alibaba
- qwen3.6-max-preview
- qwen3.6-plus
- qwen3.7-max
- qwen3.8-max-0902
- qwen3.8-max-prime
- gemini-2.5-flash
- gemini-3-flash-preview
- gemini-3.1-pro-preview
- gemini-4-argon
Meta
- muse-spark-1.1
- muse-spark-1.3
- muse-spark-open
- muse-wonder
ByteDance
- seed-2.0-code
- seed-2.0-lite
- seed-2.1-turbo
Sakana
- fugu-ultra
- fugu-ultra-v2
- sakana-namazu
Z.ai
- glm-5.2
- glm-5.3
- glm-5.3-prime
InclusionAI
- ling-2.6-1t
- ling-3.0-flash
MiniMax
- minimax-m2.5
- minimax-m3
Mistral
- mistral-large-4
- mistral-medium-3.5
Moonshot
- kimi-k2.7-code
- kimi-k3
Nvidia
- nemotron-3-ultra
- nemotron-3.5-lightning
Poolside
- laguna-s-2.1
- laguna-xs-2.1
StepFun
- step-3.7-flash
- step-5-preview
Meituan
- longcat-2.0
OpenAI
- gpt-5.4-mini
Tencent
- hy3
ThinkingMachines
- inkling
Upstage
- solar-pro4
Xiaomi
- mimo-v2.6-pro
One ReGrade context block in the agent's prompt. No model retraining. No changes to your build or CI pipeline. Tested working across 84 models from 22 labs, US and non-US.
Reduce unintended changes reaching review.
Every lab we tested produced them, all 22 labs. ReGrade cut them by over 83% on the typical model. On 28 models, across 11 labs, no unintended changes were observed in the ReGrade trials.
Reduce unintended code changes.