Skip to main content
Curtail

AI coding hallucinations

Ongoing testing

In our ongoing testing, ReGrade® reduces AI coding hallucinations by over 86%.

ReGrade records real traffic, replays it against your candidate version and reports what changed, for engineering leaders who approve releases.

An AI coding hallucination is an unintended change, addition or deletion to working code that the agent was never asked to modify.

Curtail research · 88 models, 24 labs, updated 2026-11-05

How we tested

Every AI lab we tested produced AI coding hallucinations. All 24 labs. The strongest models included.

With ReGrade AI coding hallucinations dropped to zero for these models.

A hallucination is counted by replaying each trial's saved patch offline and counting the changes that fall outside the bugs the model was asked to fix. Each change is traced back to the edit that caused it, by removing one edit at a time and replaying, so one mistake that surfaces at nine call sites counts once. On the typical model, ReGrade's patented behavioral comparison cut those changes by over 86%. In the main comparisons, no unintended changes were observed in the ReGrade trials on the 31 models below, across 14 labs. Zero observed changes is not a guarantee on future code.

  • OpenAIGPT-6 Astra, GPT-6.1 Sol, GPT-5.5, GPT-6.1 Terra, GPT-5.6 Sol, GPT-6 Sol, GPT-5.6 Terra
  • XiaomiMiMo V2.6 Pro
  • xAIGrok 4.5, Grok 3 Fast
  • MoonshotKimi K2.7 Code, Kimi K3.1 Pro
  • MetaMuse Wonder, Muse Spark 1.3, Muse Spark Open
  • SakanaFugu Ultra v2, Fugu Ultra
  • AnthropicFable 5, Opus 4.8, Opus 5, Fable 5.1, Opus 4.6
  • GoogleGemini 4 Argon, Gemini 3.1 Pro Preview
  • MiniMaxMiniMax M3.1 Flash
  • DeepSeekDeepSeek V4.1 Flash 1017
  • AlibabaQwen3.8 Max Prime, Qwen3.7 Max
  • InceptionMercury 3
  • NvidiaNemotron 3.5 Lightning
  • ReflectionReflection Beam 1.0

Every model tested, on AI coding hallucinations

Showing 10 of 88 models
Results by model, without and with ReGrade: hallucinations per trial
HallucinationsBug Fixes
Verdict
gpt-6-astra7.270−100%14.418.0OpenAI30✓
gpt-6.1-terra2.830−100%14.318.0OpenAI30✓
gpt-6.1-sol5.430−100%14.118.0OpenAI30✓
gpt-6-sol2.430−100%14.018.0OpenAI30✓
muse-wonder3.670−100%13.317.3Meta3✓
muse-spark-1.12.750.33−88%13.217.7Meta3✓
gpt-5.43.160.24−92%13.117.9OpenAI31✓
gpt-5.53.040−100%13.018.0OpenAI27✓
fable-5.11.670−100%12.718.0Anthropic30✓
gemini-4-argon2.700−100%12.718.0Google18✓

✓ improved · * got worse · N/A not scorable. Click a column to sort. Scroll sideways to see the lab and sample-count columns.

About this table

N/A: the model did not edit the file in the control run, so there is no baseline to compare against. The table lists the models with a measurement on this metric, from a corpus of 88 models, 24 labs and 2,179 agent-mode trials. The provider-pinned comparisons add 30 separately labeled trials, for 2,209 total. Min. replays/arm is the smaller eligible replay count of the control and treatment arms; individual arms can contain more runs. Bug fixes are unaided (mean bugs fixed across the model's control-arm trials) and aided (the same average over the treatment trials), so the table can be sorted by how capable a model is on its own and by how far it gets with help. Both are our own measurements. Repair and damage eligible subsets can differ, so bug-fix averages need not use every replay counted here. A dash means the model sits below the trial floor, so no figure is published for it.

The honest caveats. The main comparison results are in the table above, sortable and searchable. The provider-pinned results follow separately, with three trials per arm: July DeepSeek on Baidu reached zero observed changes and Nemotron Ultra's count rose. Those samples do not establish stable effects.

Provider-pinned comparisons

Five separate cells, 30 trials, three trials per arm. Both arms of each cell used the same provider. These batches remain separate from the historical model comparisons above and their headline statistics. Every arrow reads without ReGrade → with ReGrade.

Provider-pinned repair and causal damage measurements
ModelProvider and precisionBugs fixed / 18Causal damage per trial
deepseek-v4-flash-0423DeepInfra · fp84.33 → 16.001.33 → 0.33
deepseek-v4-flash-0731Baidu · fp85.67 → 17.002.67 to 3.33 → 0.00
nemotron-3-ultraBaseTen · fp44.33 → 15.001.33 → 1.67
inklingTogether · precision unknown8.67 → 14.333.67 → 1.67
glm-5.2Baidu · fp47.00 → 16.672.00 to 3.00 → 1.00

Higher repair scores and lower damage are better. Ranges include unresolved hunk-removal variants. Nemotron introduced slightly more damage with ReGrade. Three trials per arm do not establish stable model effects.

Works with every major coding agent and model API.

5 coding-agent CLIs · 88 models · 24 labs.

Run through Claude Code, Codex CLI, qwen-code, Kimi Code CLI, Mistral Vibe CLI.

Coding-agent CLIs5 CLIs · 28 models

Claude Code

  • claude-haiku-5-5
  • fable-5
  • fable-5.1
  • glm-5.1
  • haiku-4.5
  • haiku-4.5 (Aug 2026)
  • opus-4.6
  • opus-4.7
  • opus-4.8
  • opus-5
  • opus-5.5
  • sonnet-4.6
  • sonnet-5
  • sonnet-5.5

Codex CLI

  • gpt-5.4
  • gpt-5.5
  • gpt-5.6-luna
  • gpt-5.6-sol
  • gpt-5.6-terra
  • gpt-6-astra
  • gpt-6-luna
  • gpt-6-sol
  • gpt-6.1-sol
  • gpt-6.1-terra

qwen-code

  • qwen3-coder
  • qwen3-coder-plus

Kimi Code CLI

  • kimi-k26

Mistral Vibe CLI

  • devstral-2
Models tested via API, by lab60 models · 23 labs

xAI

  • grok-3-fast
  • grok-3-mini
  • grok-4-1-fast-non-reasoning
  • grok-4.20-0309-reasoning
  • grok-4.3
  • grok-4.5
  • grok-4.7
  • grok-build-0.1

DeepSeek

  • deepseek-v4-flash-0423
  • deepseek-v4-flash-0731
  • deepseek-v4-pro (Sep 2026)
  • deepseek-v4-pro-0423
  • deepseek-v4-pro-0813
  • deepseek-v4.1-flash
  • deepseek-v4.1-flash-1017

Alibaba

  • qwen3.6-max-preview
  • qwen3.6-plus
  • qwen3.7-max
  • qwen3.8-max-0902
  • qwen3.8-max-prime

Google

  • gemini-2.5-flash
  • gemini-3-flash-preview
  • gemini-3.1-pro-preview
  • gemini-4-argon

Meta

  • muse-spark-1.1
  • muse-spark-1.3
  • muse-spark-open
  • muse-wonder

ByteDance

  • seed-2.0-code
  • seed-2.0-lite
  • seed-2.1-turbo

MiniMax

  • minimax-m2.5
  • minimax-m3
  • minimax-m3.1-flash

Moonshot

  • kimi-k2.7-code
  • kimi-k3
  • kimi-k3.1-pro

Sakana

  • fugu-ultra
  • fugu-ultra-v2
  • sakana-namazu

Z.ai

  • glm-5.2
  • glm-5.3
  • glm-5.3-prime

InclusionAI

  • ling-2.6-1t
  • ling-3.0-flash

Mistral

  • mistral-large-4
  • mistral-medium-3.5

Nvidia

  • nemotron-3-ultra
  • nemotron-3.5-lightning

StepFun

  • step-3.7-flash
  • step-5-preview

Inception

  • mercury-3

Meituan

  • longcat-2.0

OpenAI

  • gpt-5.4-mini

Poolside

  • laguna-xs-2.1

Reflection

  • reflection-beam-1.0

Tencent

  • hy3

ThinkingMachines

  • inkling

Upstage

  • solar-pro4

Xiaomi

  • mimo-v2.6-pro

One ReGrade context block in the agent's prompt. No model retraining. No changes to your build or CI pipeline. Tested working across 88 models from 24 labs, US and non-US.

Reduce unintended changes reaching review.

Every lab we tested produced them, all 24 labs. ReGrade cut them by over 86% on the typical model. On 31 models, across 14 labs, no unintended changes were observed in the ReGrade trials.

Contact us on LinkedIn