More bugs fixed
OngoingIn our ongoing testing, ReGrade helps AI coding agents fix 133% more bugs.
Measured on the typical model, across 68 models from 22 labs. 65 of the 67 we scored fixed more silent regressions with ReGrade.
Curtail research · 68 models, 22 labs, updated 2026-09-18
The AFRL study measured this across 17 models in April to May 2026. These figures cover 68 models. Read the study as published.
Every model tested, on bugs fixed
| Bug Fixes | ||||||
|---|---|---|---|---|---|---|
| Verdict | ||||||
| qwen3-coder-plus | 2.00 | 11.8 | +488% | Alibaba | 7 | ✓ |
| laguna-xs-2.1 | 1.80 | 10.3 | +474% | Poolside | 30 | ✓ |
| haiku-4.5 (Aug 2026) | 2.70 | 14.5 | +437% | Anthropic | 20 | ✓ |
| minimax-m2.5 | 2.10 | 10.9 | +427% | MiniMax | 60 | ✓ |
| haiku-4.5 | 2.50 | 12.8 | +413% | Anthropic | 60 | ✓ |
| seed-2.0-lite | 2.90 | 14.7 | +408% | ByteDance | 60 | ✓ |
| nemotron-3-ultra | 3.00 | 13.8 | +360% | Nvidia | 10 | ✓ |
| ling-2.6-1t | 1.60 | 7.00 | +338% | InclusionAI | 10 | ✓ |
| qwen3.6-plus | 4.00 | 16 | +300% | Alibaba | 7 | ✓ |
| grok-4-1-fast-non-reasoning | 2.50 | 9.70 | +295% | xAI | 21 | ✓ |
| ling-3.0-flash | 2.90 | 11.3 | +294% | InclusionAI | 29 | ✓ |
| glm-5.1 | 4.50 | 17.6 | +292% | Z.ai | 60 | ✓ |
| grok-4.20-0309-reasoning | 3.20 | 12.3 | +284% | xAI | 21 | ✓ |
| nemotron-3.5-lightning | 1.40 | 5.30 | +281% | Nvidia | 8 | ✓ |
| gpt-5.4-mini | 3.60 | 13.2 | +267% | OpenAI | 10 | ✓ |
| kimi-k26 | 4.40 | 16 | +261% | Moonshot | 13 | ✓ |
| deepseek-v4-flash-0731 | 4.80 | 16.7 | +252% | DeepSeek | 22 | ✓ |
| step-3.7-flash | 3.40 | 11 | +221% | StepFun | 60 | ✓ |
| grok-4.3 | 2.00 | 6.30 | +217% | xAI | 6 | ✓ |
| sonnet-5 | 5.50 | 17.4 | +216% | Anthropic | 60 | ✓ |
| kimi-k2.7-code | 5.00 | 15.7 | +213% | Moonshot | 6 | ✓ |
| deepseek-v4-pro | 5.30 | 16.6 | +211% | DeepSeek | 8 | ✓ |
| qwen3-coder | 1.70 | 5.00 | +192% | Alibaba | 15 | ✓ |
| solar-pro4 | 3.80 | 10.6 | +179% | Upstage | 10 | ✓ |
| opus-4.6 | 7.00 | 17.7 | +152% | Anthropic | 6 | ✓ |
| glm-5.2 | 6.70 | 16.6 | +150% | Z.ai | 39 | ✓ |
| qwen3.6-max-preview | 5.70 | 13.8 | +144% | Alibaba | 8 | ✓ |
| devstral-2 | 5.40 | 13 | +142% | Mistral | 60 | ✓ |
| sonnet-4.6 | 7.30 | 17.6 | +141% | Anthropic | 87 | ✓ |
| minimax-m3 | 7.30 | 17.7 | +141% | MiniMax | 6 | ✓ |
| sakana-namazu | 5.00 | 12 | +140% | Sakana | 7 | ✓ |
| hy3 | 6.00 | 14 | +133% | Tencent | 9 | ✓ |
| mistral-medium-3.5 | 6.40 | 14.9 | +133% | Mistral | 20 | ✓ |
| qwen3.7-max | 7.30 | 16.7 | +127% | Alibaba | 6 | ✓ |
| gpt-5.6-luna | 7.50 | 16.8 | +123% | OpenAI | 60 | ✓ |
| deepseek-v4-pro-0813 | 6.80 | 13.6 | +100% | DeepSeek | 30 | ✓ |
| opus-4.8 | 9.10 | 17.9 | +96% | Anthropic | 60 | ✓ |
| glm-5.3 | 8.70 | 16.3 | +89% | Z.ai | 6 | ✓ |
| deepseek-v4-flash | 3.80 | 7.00 | +84% | DeepSeek | 8 | ✓ |
| gemini-3-flash-preview | 9.60 | 17.6 | +83% | 47 | ✓ | |
| fugu-ultra | 10.2 | 18 | +77% | Sakana | 10 | ✓ |
| gemini-3.1-pro-preview | 8.20 | 14.5 | +76% | 8 | ✓ | |
| opus-4.7 | 10.6 | 17.8 | +67% | Anthropic | 84 | ✓ |
| longcat-2.0 | 6.70 | 11 | +65% | Meituan | 6 | ✓ |
| inkling | 9.30 | 15.3 | +64% | ThinkingMachines | 6 | ✓ |
| grok-4.5 | 11.2 | 18 | +60% | xAI | 8 | ✓ |
| fable-5 | 11.5 | 18 | +57% | Anthropic | 32 | ✓ |
| muse-spark-1.3 | 11.4 | 17.8 | +56% | Meta | 10 | ✓ |
| hy4-preview | 11.3 | 17.7 | +56% | Tencent | 6 | ✓ |
| opus-5 | 11.9 | 18 | +52% | Anthropic | 60 | ✓ |
| gpt-5.6-terra | 12 | 17.9 | +49% | OpenAI | 60 | ✓ |
| fable51 | 12.7 | 18 | +42% | Anthropic | 6 | ✓ |
| muse-spark-1.2 | 12 | 17 | +42% | Meta | 6 | ✓ |
| qwen3.8-max-0902 | 12.6 | 17.6 | +40% | Alibaba | 10 | ✓ |
| kimi-k3 | 12.3 | 17 | +38% | Moonshot | 6 | ✓ |
| gpt-5.5 | 13.3 | 18 | +36% | OpenAI | 67 | ✓ |
| muse-spark-1.1 | 13.2 | 17.7 | +33% | Meta | 7 | ✓ |
| gpt-5.6-sol | 13.9 | 18 | +30% | OpenAI | 60 | ✓ |
| gpt-5.4 | 13.9 | 17.9 | +29% | OpenAI | 79 | ✓ |
| gpt-6-astra | 15 | 18 | +20% | OpenAI | 6 | ✓ |
| gemini-2.5-flash | 5.30 | 5.60 | +6% | 15 | ✓ | |
| grok-3-mini | 1.90 | 1.90 | 0% | xAI | 16 | no change |
| grok-3-fast | 6.30 | 5.50 | −13% | xAI | 21 | * |
| gemini-3.5-flash | did not edit | 16.7 | N/A | 6 | no baseline | |
| gemini-3.6-flash | did not edit | 8.20 | N/A | 11 | no baseline | |
| gpt-oss-20b | did not edit | 3.80 | N/A | OpenAI | 10 | no baseline |
| laguna-s-2.1 | did not edit | 10 | N/A | Poolside | 6 | no baseline |
✓ improved · * got worse · N/A: the model did not edit the file in the control run, so there is no baseline to compare against. Click a column to sort. Measured over 68 models, 22 labs, 1,571 agent-mode trials. Bug fixes are unaided (mean bugs fixed across the model's control-arm trials) and aided (the same average over the treatment trials), so the table can be sorted by how capable a model is on its own and by how far it gets with help. Both are our own measurement from the same trials as the rest of this table, not a third party's score. — means the model sits below the trial floor, so no figure is published for it.
How we tested.
ReGrade is a behavioral-diff context block your AI coding agent reads alongside its existing prompt. The block shows the agent how the new code's runtime behavior differs from the old, the way a human checks for regressions at code review. No model retraining. No changes to your build or CI pipeline.
How we measured it. At Curtail® we tested 68 AI models on one task: fix 18 real bugs in a working 300,000 line Python service. Every model ran that task twice. On the first run the model worked blind, the way a coding agent normally works today. On the second run we gave the model ReGrade's patented behavioral comparison, which runs the same live traffic against both versions of the service and shows the model exactly what its last changes did to the running system. 1,571 agent-mode trials across 68 AI models from 22 labs, US and non-US, from frontier flagships down to free tiers.
What counts as an AI coding hallucination. An unintended change, addition or deletion to working code the agent was never asked to modify. The code is not what hallucinates; the AI writing it is. We measure it by replaying the agent's patch against the original behavior and counting what changed that nobody requested. Each change is traced back to the edit that caused it, by removing one edit at a time and replaying, so one mistake surfacing at nine call sites counts once.
68
AI models
22
labs
1,571
agent-mode trials
18
known bugs
300,000-line Python
codebase
-73%
AI coding hallucinations
🕵️ It catches what your tests can't.
Functional tests check status codes and required fields. They routinely miss the behavioral drift that actually breaks production: a CORS header that quietly changed, a list that's now an object, a timestamp format that broke a downstream consumer. ReGrade compares runtime behavior directly, so the agent sees the drift the test suite waved through.
35%
of AI patches that passed every functional test still silently broke behavior
191 of 548 test-passing patches, across 14 of 26 tasks
0 → 9
multi-file bug sites fixed, weaker models fixed 0 of 9 on their own
with ReGrade, agents clear all 9 sites of a cross-file regression
18
classes of silent drift planted, headers, encoding, ordering, response shape
the behavioral defects that pass type-checks, lint, and existing tests
Your test suite says “green.” ReGrade says “but the API’s behavior drifted.” That gap is exactly where production incidents come from.
Measured on BaxBench and the multi-site experiment in July 2026. These three figures come from that experiment rather than from the running corpus, so they do not move as more models are tested.
🔎 Better answers, and better detection.
Finding a bug is only half the job, a reviewer still has to understand and trust the fix. Even when a model already spots a regression, ReGrade’s behavioral evidence helps it explain exactly what changed and where.
2×
more precise bug explanations on a production codebase (Ghost CMS)
explanation precision rose from 0.40 to 0.91
9 of 10
models explained the bug better on a 291,000-line codebase (NetBox)
+3.2 points on a 20-point rubric, even where the model already found it
The difference between “something looks off” and a root-cause a human can act on with confidence.
In collaboration with AFRL · figures as published
Distribution (A) Approved for public release; distribution is unlimited; AFRL-2026-3793, AUG 2026
Is it the evidence, or just the extra words?
The obvious objection to the whole study is that any five thousand tokens would help, and the behavioral report just happens to be five thousand tokens. So we pre-registered a test of exactly that: the same agents, the same bugs, but the report swapped for pseudo-random tokens of the same length, and again for unrelated source code of the same length.
| Model | No extra contextNothing | Random tokensRandom | Unrelated sourceSource | ReGradeReGrade |
|---|---|---|---|---|
| Haiku 4.5 | 2.4 | 0.8 | 1.9 | 12.5 |
| Sonnet 4.6 | 5.5 | 5.9 | 5.3 | 17.7 |
| Opus 4.7 | 9.1 | 9.2 | 8.7 | 18.0 |
| GPT-5.4 | 13.3 | 14.1 | 12.4 | 18.0 |
| GPT-5.5 | 12.3 | 15.2 | 14.1 | 18.0 |
Bugs fixed out of 18, mean of ten trials per model per arm.
The rescue does not survive the substitution. Padding of either kind lands within about a bug of no extra context at all, while the behavioral report lands eight bugs above it. All three pre-registered hypotheses reject at p below 1e-4.
- +8.32 bugs against no extra context (95% CI 7.32 to 9.32)
- +7.80 bugs against random tokens of the same length (95% CI 6.66 to 8.90)
- +8.36 bugs against unrelated source of the same length (95% CI 7.40 to 9.30)
Two models are worth reading closely. Haiku gets worse under random tokens than with nothing at all, 0.8 against 2.4, because noise costs it attention it does not have to spare. The OpenAI models move the other way: GPT-5.5 gains from padding of either kind, 15.2 and 14.1 against a control of 12.3, so some of the lift there is volume rather than signal. The behavioral report still beats both, at 18.0.
Get more fixed from the agents you already run.
65 of the 67 models we scored fixed more silent regressions with ReGrade, on the traffic a team already has.
Fix more bugs with the agents you run.