Skip to main content
Curtail

AI coding hallucinations

Ongoing

In our ongoing testing, ReGrade reduces AI coding hallucinations by over 73%.

An AI coding hallucination is an unintended change, addition or deletion to working code that the agent was never asked to modify.

Curtail research · 68 models, 22 labs, updated 2026-09-18

Where AI coding hallucinations stopped.

A hallucination is counted by replaying each trial's saved patch offline and counting the changes that fall outside the bugs the model was asked to fix. Each change is traced back to the edit that caused it, by removing one edit at a time and replaying, so one mistake that surfaces at nine call sites counts once. On the typical model, ReGrade's patented behavioral comparison cut those changes by over 73%. On the 17 models below, across 10 labs, it stopped them completely.

  • OpenAIgpt-6-astra, gpt-5.5, gpt-5.6-sol
  • xAIgrok-4.5, grok-3-fast
  • Metamuse-spark-1.3
  • Anthropicfable-5, fable51, opus-4.8, opus-5, opus-4.6
  • Moonshotkimi-k2.7-code
  • Sakanafugu-ultra
  • Googlegemini-3.1-pro-preview
  • Alibabaqwen3.7-max
  • Tencenthy4-preview
  • Nvidianemotron-3.5-lightning

Every AI lab we tested produced AI coding hallucinations. All 22 labs. The strongest models included.

The honest caveats. On 1 model hallucinations rose with ReGrade: deepseek-v4-pro-0813. A further 4 produced none unaided because they did not edit the file at all, so they are reported as N/A rather than counted as successes. Every model we tested is in the table below, sortable and searchable.

Every model tested, on AI coding hallucinations

68 models · 22 labs
HallucinationsBug Fixes
Verdict
gpt-6-astra100−100%15.018.0OpenAI3✓
gpt-5.42.890.22−92%13.917.9OpenAI35✓
gpt-5.6-sol2.330−100%13.918.0OpenAI30✓
gpt-5.52.880−100%13.318.0OpenAI33✓
muse-spark-1.12.750.33−88%13.217.7Meta3✓
fable512.000−100%12.718.0Anthropic3✓
qwen3.8-max-09021.400.40−71%12.617.6Alibaba5✓
kimi-k35.330.33−94%12.317.0Moonshot3✓
muse-spark-1.24.000.67−83%12.017.0Meta3✓
gpt-5.6-terra2.240.07−97%12.017.9OpenAI29✓
opus-51.830−100%11.918.0Anthropic30✓
fable-53.310−100%11.518.0Anthropic16✓
muse-spark-1.33.400−100%11.417.8Meta5✓
hy4-preview1.670−100%11.317.7Tencent3✓
grok-4.53.750−100%11.218.0xAI4✓
opus-4.71.660.03−98%10.617.8Anthropic38✓
fugu-ultra2.400−100%10.218.0Sakana5✓
gemini-3-flash-preview0.500.29−43%9.617.6Google6✓
inkling3.671.67−54%9.315.3ThinkingMachines3✓
opus-4.81.970−100%9.117.9Anthropic21✓
glm-5.32.001.00−50%8.716.3Z.ai3✓
gemini-3.1-pro-preview1.670−100%8.214.5Google3✓
gpt-5.6-luna1.470.70−52%7.516.8OpenAI30✓
qwen3.7-max1.670−100%7.316.7Alibaba3✓
sonnet-4.60.970.03−97%7.317.6Anthropic38✓
minimax-m31.330.33−75%7.317.7MiniMax3✓
opus-4.61.330−100%7.017.7Anthropic3✓
deepseek-v4-pro-08130.851.00+18%6.813.6DeepSeek13*
longcat-2.03.000.33−89%6.711.0Meituan3✓
glm-5.21.850.74−60%6.716.6Z.ai19✓
mistral-medium-3.54.002.10−47%6.414.9Mistral10✓
grok-3-fast1.330−100%6.35.5xAI6✓
hy32.751.20−56%6.014.0Tencent4✓
qwen3.6-max-preview2.000.25−87%5.713.8Alibaba3✓
sonnet-52.000.27−87%5.517.4Anthropic30✓
devstral-24.400.90−79%5.413.0Mistral30✓
deepseek-v4-pro1.000.50−50%5.316.6DeepSeek3✓
gemini-2.5-flash3.001.67−44%5.35.6Google3✓
kimi-k2.7-code3.000−100%5.015.7Moonshot3✓
sakana-namazu1.750.20−89%5.012.0Sakana4✓
deepseek-v4-flash-07310.910.60−34%4.816.7DeepSeek10✓
glm-5.11.400.53−62%4.517.6Z.ai30✓
kimi-k261.570.50−68%4.416.0Moonshot7✓
qwen3.6-plus0.670.33−50%4.016.0Alibaba3✓
deepseek-v4-flash2.000.33−83%3.87.0DeepSeek3✓
solar-pro41.201.00−17%3.810.6Upstage4✓
gpt-5.4-mini1.600.60−62%3.613.2OpenAI5✓
step-3.7-flash2.861.00−65%3.411.0StepFun26✓
grok-4.20-0309-reasoning7.005.60−20%3.212.3xAI10✓
nemotron-3-ultra1.601.40−12%3.013.8Nvidia5✓
seed-2.0-lite2.500.43−83%2.914.7ByteDance16✓
ling-3.0-flash1.001.000%2.911.3InclusionAI14no change
haiku-4.5 (Aug 2026)1.701.60−6%2.714.5Anthropic10✓
haiku-4.52.301.23−46%2.512.8Anthropic30✓
grok-4-1-fast-non-reasoning2.360.71−70%2.59.7xAI14✓
minimax-m2.51.201.13−6%2.110.9MiniMax30✓
qwen3-coder-plus1.000.33−67%2.011.8Alibaba3✓
grok-4.34.001.33−67%2.06.3xAI3✓
grok-3-mini0.670.33−50%1.91.9xAI3✓
laguna-xs-2.11.801.60−11%1.810.3Poolside15✓
qwen3-coder1.330.67−50%1.75.0Alibaba3✓
ling-2.6-1t1.801.75−3%1.67.0InclusionAI4✓
nemotron-3.5-lightning1.250−100%1.45.3Nvidia4✓
gemini-3.5-flashdid not edit0.33N/A0.016.7Google3no baseline
gemini-3.6-flashdid not edit0.17N/A0.08.2Google6no baseline
gpt-oss-20bdid not edit–N/A0.03.8OpenAI5no baseline
laguna-s-2.1did not edit0.33N/A0.010.0Poolside3no baseline
kat-coder-pro-v2.50.750.25−67%——Kuaishou4✓

✓ improved · * got worse · N/A: the model did not edit the file in the control run, so there is no baseline to compare against. Click a column to sort. Measured over 68 models, 22 labs, 1,571 agent-mode trials. Bug fixes are unaided (mean bugs fixed across the model's control-arm trials) and aided (the same average over the treatment trials), so the table can be sorted by how capable a model is on its own and by how far it gets with help. Both are our own measurement from the same trials as the rest of this table, not a third party's score. — means the model sits below the trial floor, so no figure is published for it.

🛠️ Works with every major coding agent and model API.

5 coding-agent CLIs · 68 models · 22 labs.

Coding-agent CLIs5 CLIs · 21 models

Claude Code

  • fable-5
  • fable51
  • glm-5.1
  • haiku-4.5
  • haiku-4.5 (Aug 2026)
  • opus-4.6
  • opus-4.7
  • opus-4.8
  • opus-5
  • sonnet-4.6
  • sonnet-5

Codex CLI

  • gpt-5.4
  • gpt-5.5
  • gpt-5.6-luna
  • gpt-5.6-sol
  • gpt-5.6-terra
  • gpt-6-astra

qwen-code

  • qwen3-coder
  • qwen3-coder-plus

Kimi Code CLI

  • kimi-k26

Mistral Vibe CLI

  • devstral-2
Models tested via API, by lab47 models · 21 labs

xAI

  • grok-3-fast
  • grok-3-mini
  • grok-4-1-fast-non-reasoning
  • grok-4.20-0309-reasoning
  • grok-4.3
  • grok-4.5

Google

  • gemini-2.5-flash
  • gemini-3-flash-preview
  • gemini-3.1-pro-preview
  • gemini-3.5-flash
  • gemini-3.6-flash

Alibaba

  • qwen3.6-max-preview
  • qwen3.6-plus
  • qwen3.7-max
  • qwen3.8-max-0902

DeepSeek

  • deepseek-v4-flash
  • deepseek-v4-flash-0731
  • deepseek-v4-pro
  • deepseek-v4-pro-0813

Meta

  • muse-spark-1.1
  • muse-spark-1.2
  • muse-spark-1.3

InclusionAI

  • ling-2.6-1t
  • ling-3.0-flash

MiniMax

  • minimax-m2.5
  • minimax-m3

Moonshot

  • kimi-k2.7-code
  • kimi-k3

Nvidia

  • nemotron-3-ultra
  • nemotron-3.5-lightning

OpenAI

  • gpt-5.4-mini
  • gpt-oss-20b

Poolside

  • laguna-s-2.1
  • laguna-xs-2.1

Sakana

  • fugu-ultra
  • sakana-namazu

Tencent

  • hy3
  • hy4-preview

Z.ai

  • glm-5.2
  • glm-5.3

ByteDance

  • seed-2.0-lite

Kuaishou

  • kat-coder-pro-v2.5

Meituan

  • longcat-2.0

Mistral

  • mistral-medium-3.5

StepFun

  • step-3.7-flash

ThinkingMachines

  • inkling

Upstage

  • solar-pro4

One ReGrade context block in the agent's prompt. No model retraining. No changes to your build or CI pipeline. Tested working across 68 models from 22 labs, US and non-US.

🌍 The newest labs, too.

The behavioral-diff effect is not a quirk of the big US providers. We ran the same test on the newest releases from five more labs, at 30 trials per model, and the rescue held every time.

LabModelWithoutWithΔ
ByteDanceSeed-2.0-Lite2.9 / 1814.7 / 18+408%
MiniMaxM2.52.1 / 1810.9 / 18+427%
ZhipuGLM-5.14.5 / 1817.6 / 18+292%
StepFunStep 3.7-flash3.4 / 1811.0 / 18+221%
MistralDevstral 25.4 / 1813.0 / 18+142%

68 models across 22 labs, US and non-US, from frontier flagships down to free tiers. 65 of the 67 we could measure fixed more silent regressions with ReGrade.

Stop hallucinations reaching your codebase.

Every lab we tested produced them, all 22 labs. ReGrade cut them by over 73% on the typical model. On 17 models, across 10 labs, it stopped them completely.

Ask us how we can helpTry For Free NowContact us on LinkedIn
Ask usTry For Free