Cost reduction
Ongoing testingIn our ongoing testing, ReGrade® cuts LLM API cost per bug fixed by over 58%.
ReGrade records real traffic, replays it against your candidate version and reports what changed, for engineering leaders who approve releases.
Across 75 models in Curtail’s current corpus, the typical model cut its LLM API cost per bug fixed by 58.5% with ReGrade. 72 of the 75 got cheaper per bug fixed. LLM API cost per bug fixed is the model API spend divided by the bugs the agent actually eliminated, at the rate each provider billed, or its published price where no bill exists. Subscription fees, deployment work and recording/replay infrastructure are excluded.
Curtail research · 84 models, 22 labs, updated 2026-10-24
The AFRL study measured this across 17 models in April to May 2026, where the median model cut cost per bug fixed by 42%, priced at each provider’s list price. These figures cover the 75 models with a cost per bug fixed both with and without ReGrade, priced at the rate each provider billed or, where no bill exists, its published price. Read the study as published.
Adding the behavioral report raises what a run costs, and the agent finds enough more that the cost per bug fixed falls.
Models with LLM API cost measurements
75 models have LLM API cost measurements, and 75 of them have a cost per bug fixed both with and without ReGrade. The table excludes models without cost measurements; the full corpus contains 84 models.
| Cost | Bug Fixes | |||||||
|---|---|---|---|---|---|---|---|---|
| Verdict | ||||||||
| deepseek-v4.1-flash | $0.1047 | $0.0037 | −96% | 8.3 | 17.3 | DeepSeek | 6 | ✓ |
| deepseek-v4.1-flash-1017 | $0.0782 | $0.0038 | −95% | 7.7 | 18.0 | DeepSeek | 6 | ✓ |
| ling-2.6-1t | $0.0873 | $0.0064 | −93% | 1.6 | 7.0 | InclusionAI | 10 | ✓ |
| deepseek-v4-pro-0813 | $0.3235 | $0.0273 | −92% | 3.2 | 13.5 | DeepSeek | 25 | ✓ |
| qwen3.6-plus | $0.0609 | $0.0238 | −61% | 4.0 | 16.0 | Alibaba | 6 | ✓ |
| grok-4.20-0309-reasoning | $0.2735 | $0.1098 | −60% | 3.2 | 11.7 | xAI | 20 | ✓ |
| glm-5.3-prime | $0.1738 | $0.0722 | −58% | 10.0 | 16.7 | Z.ai | 6 | ✓ |
| ling-3.0-flash | $0.0094 | $0.0039 | −58% | 1.9 | 11.3 | InclusionAI | 27 | ✓ |
| gpt-6-astra | $0.0465 | $0.0441 | −5% | 14.7 | 18.0 | OpenAI | 6 | ✓ |
| qwen3-coder | $0.0347 | $0.0377 | +9% | 4.7 | 12.5 | Alibaba | 20 | * |
| mistral-medium-3.5 | $0.0503 | $0.0605 | +20% | 6.4 | 14.9 | Mistral | 20 | * |
| gemini-3.1-pro-preview | $0.0902 | $0.1134 | +26% | 10.5 | 16.0 | 20 | * | |
✓ improved · * got worse · N/A not scorable. Click a column to sort. Scroll sideways to see the lab and sample-count columns.
About this table
N/A: the model fixed no bugs without ReGrade, so there is no cost per bug to compare against. The table lists the models with a measurement on this metric, from a corpus of 84 models, 22 labs and 1,633 agent-mode trials. The provider-pinned comparisons add 30 separately labeled trials, for 1,663 total. Runs, both arms counts control and treatment runs together. Bug fixes are unaided (mean bugs fixed across the model's control-arm trials) and aided (the same average over the treatment trials), so the table can be sorted by how capable a model is on its own and by how far it gets with help. Both are our own measurements. Repair and damage eligible subsets can differ, so bug-fix averages need not use every replay counted here. A dash means the model sits below the trial floor, so no figure is published for it.
Three recommended models, ranked by LLM API cost per bug fixed.
Measured on DriftBench, our open-source benchmark of 18 seeded regressions, among models fixing at least 16 of 18 bugs with ReGrade.
Ranked from Curtail's current corpus of 84 models, by LLM API cost per bug fixed with ReGrade among those fixing at least 16 of 18. This list moves as later testing continues.
🥇
claude-haiku-5-5
17.7 / 18
bugs fixed
$0.0017
LLM API cost per bug
🥈
deepseek-v4-flash-0731
16.7 / 18
bugs fixed
$0.0029
LLM API cost per bug
🥉
deepseek-v4.1-flash
17.3 / 18
bugs fixed
$0.0037
LLM API cost per bug
44 more model and ReGrade combinations clear the same threshold, at least 16 of 18 bugs fixed. Models from multiple providers met this threshold.
These figures measure LLM API cost per bug fixed. ReGrade subscription fees, recording and replay infrastructure, deployment work and human review are excluded. Use your own service to measure total cost.
See ReGrade subscription pricingEvaluate LLM API savings on your service.
Across 75 models, 72 got cheaper per bug fixed, and the typical model cut it by 58.5%. Ask us what that looks like on your stack.
Evaluate LLM API savings on your service.