AI-Generated Code Safety and Security
A 17-model, 6-provider evaluation, in collaboration with the Air Force Research Laboratory and cleared for public release. The figures on this page are the study as published and do not change as later testing continues.
Every one of the 17 models tested fixed more bugs with ReGrade. The median model fixed 152% more, at a median 42% lower cost per bug fixed.
Download the study
Full paper and executive summary
Executive summary (same content as the downloadable PDF)
Companion overview for the full paper “AI-Generated Code Safety and Security — Network Comparison Application Security Testing (NCAST): A 17-Model, 6-Provider Evaluation,” a joint study conducted by Curtail and the Air Force Research Laboratory (AFRL). See the full paper for methodology and detailed results.
Curtail® ReGrade® compares network traffic between two versions of a software service and reports what changed. Testing across 17 models from 6 AI labs, ReGrade provides AI coding agents deterministic data that increases the number of bugs fixed for all models, and lowers the cost per bug fixed for most models. The two tables below report both effects model by model.
17 models · 6 AI labs · 18 silent regressions (bugs)
+152% median more bugs fixed · -42% median lower cost per bug
“ReGrade is essential for creating higher quality and safer AI-generated code. ReGrade identifies and corrects significantly more bugs at a lower cost than AI by itself. Every one of the models tested in the study found and fixed more bugs with ReGrade than without it.”
— Frank Huerta, Chief Executive Officer, Curtail, Inc.
ReGrade helps AI coding agents fix more bugs (of 18 unknown bugs)
| Provider | Model | Mean Bugs Fixed Control (Without ReGrade) | Mean Bugs Fixed Treatment (With ReGrade) | Change |
|---|---|---|---|---|
| Anthropic | Haiku 4.5 | 1.3 / 18 | 12.3 / 18 | +825%* |
| Anthropic | Sonnet 4.6 | 5.0 / 18 | 18.0 / 18 | +260% |
| Anthropic | Opus 4.7 | 8.3 / 18 | 18.0 / 18 | +117% |
| Anthropic | Opus 4.6 | 7.0 / 18 | 17.7 / 18 | +152% |
| OpenAI | GPT-5.4 | 13.3 / 18 | 17.7 / 18 | +33% |
| OpenAI | GPT-5.5 | 13.3 / 18 | 18.0 / 18 | +35% |
| Gemini 3 Flash Preview | 9.7 / 18 | 17.7 / 18 | +82% | |
| Gemini 3.1 Pro Preview | 8.7 / 18 | 14.3 / 18 | +64% | |
| xAI | Grok 4.20 Reasoning | 3.7 / 18 | 15.7 / 18 | +324% |
| xAI | Grok 3 Fast | 2.0 / 18 | 8.0 / 18 | +300% |
| xAI | Grok 3 Mini | 1.0 / 18 | 2.7 / 18 | +167%* |
| Alibaba | Qwen3.6-Plus | 4.0 / 18 | 16.0 / 18 | +300% |
| Alibaba | Qwen3.6-Max-Preview | 5.7 / 18 | 12.8 / 18 | +125% |
| Alibaba | Qwen3-Coder-Plus | 2.0 / 18 | 12.0 / 18 | +500% |
| Alibaba | Qwen3-Coder | 4.0 / 18 | 9.0 / 18 | +125% |
| DeepSeek | DeepSeek V4 Pro | 5.3 / 18 | 16.2 / 18 | +206% |
| DeepSeek | DeepSeek V4 Flash | 3.0 / 18 | 7.0 / 18 | +133% |
Data from Table 7.1 in “Section 7.3 Primary Results” of the full paper.
Across all 17 models, ReGrade’s analysis helped AI coding agents fix more bugs, with a median increase of 152%. And with ReGrade, 13/17 models fixed over 100% more bugs.
* The control and treatment columns in the table are rounded. Because the change percentage is calculated from the unrounded trial numbers, dividing the figures shown does not reproduce the exact percentage in the two starred cases.
Cost reduction per bug fixed with ReGrade
| Provider | Model | Mean Cost Per Bug Control (Without ReGrade) | Mean Cost Per Bug Treatment (With ReGrade) | Change |
|---|---|---|---|---|
| Anthropic | Haiku 4.5 | $0.377 | $0.058 | -85% |
| Anthropic | Sonnet 4.6 | $0.126 | $0.092 | -27% |
| Anthropic | Opus 4.7 | $0.257 | $0.199 | -23% |
| Anthropic | Opus 4.6 | $0.294 | $0.097 | -67% |
| OpenAI | GPT-5.4 | $0.020 | $0.015 | -25% |
| OpenAI | GPT-5.5 | $0.156 | $0.091 | -42% |
| Gemini 3 Flash Preview | $0.015 | $0.008 | -47% | |
| Gemini 3.1 Pro Preview | $0.118 | $0.136 | +15% | |
| xAI | Grok 4.20 Reasoning | $0.235 | $0.077 | -67% |
| xAI | Grok 3 Fast | $0.105 | $0.060 | -43% |
| xAI | Grok 3 Mini | $0.010 | $0.007 | -30% |
| Alibaba | Qwen3.6-Plus | $0.060 | $0.024 | -60% |
| Alibaba | Qwen3.6-Max-Preview | $0.333 | $0.036 | -89% |
| Alibaba | Qwen3-Coder-Plus | $0.070 | $0.032 | -54% |
| Alibaba | Qwen3-Coder | $0.015 | $0.026 | +73% |
| DeepSeek | DeepSeek V4 Pro | $0.011 | $0.017 | +55% |
| DeepSeek | DeepSeek V4 Flash | $0.007 | $0.004 | -41%* |
Data from Table 7.5 in “Section 7.7 Cost Per Bug Fixed” of the full paper.
14 of 17 models cost less per bug fixed with ReGrade, with a median cost reduction of 42%.
We measured the cost as API tokens spent per bug fixed, at each provider’s list price when the sample was taken. The full paper reports the per-model detail and how cost per run differs from cost per bug fixed.
* The control and treatment columns in the table are rounded. Because the change percentage is calculated from the unrounded trial numbers, dividing the figures shown does not reproduce the exact percentage in the one starred case.
Implications of using ReGrade
ReGrade’s analysis helped every model in this study fix more bugs, and it lowered the cost per bug fixed for most of them. The full paper reports the per-model and per-bug detail. ReGrade fits the way teams already work: it runs in CI and on a developer’s machine, and it records traffic from the most common sources, whether manual browsing or an automated test suite.
Testing methodology in brief
- Models: 17 from 6 AI labs: Anthropic (Haiku 4.5, Sonnet 4.6, Opus 4.7, Opus 4.6), OpenAI (GPT-5.4, GPT-5.5), Google (Gemini 3 Flash Preview, Gemini 3.1 Pro Preview), xAI (Grok 3 Mini, Grok 3 Fast, Grok 4.20 Reasoning), Alibaba (Qwen3.6-Plus, Qwen3.6-Max-Preview, Qwen3-Coder-Plus, Qwen3-Coder), DeepSeek (V4 Pro, V4 Flash).
- Codebases: The paper tested models using BaxBench, DriftBench (~300,000-line padded codebase with 18 planted bugs and 3 multi-site bugs created by Curtail, available at github.com/Curtail-Inc/DriftBench), Ghost CMS, and NetBox (~291,000-line production system).
- Setup: an AI coding agent was given shell access to each codebase and asked to fix planted bugs, once without ReGrade’s analysis (control) and once with it (treatment). The reference version of the code was withheld from the agent in both arms, so it had to find the bugs rather than compare against a known-good copy.
- Trials: each model and benchmark combination was run 3 times unless otherwise noted in the full paper, and results are reported as the trial mean out of 18 possible bugs. Evaluation ran from April to May 2026.
Read the full paper
- Full paper: 81 pages, 31 tables, 6 figures. Available as a PDF download at the top of this page.
- DriftBench public release: the benchmark, the per-trial data and the analysis scripts, MIT licensed and public, at github.com/Curtail-Inc/DriftBench.
- ReGrade is Curtail’s commercial implementation of NCAST. The full paper calls the method Network Comparison Application Security Testing (NCAST). Where the paper says NCAST, this summary says ReGrade.
Disclosure
The corresponding authors are employed by Curtail, which developed and patented the technology evaluated in the study and has a financial interest in its adoption. The materials in the study are public so the results can be replicated independently. The study was performed under a Cooperative Research and Development Agreement (CRADA) between Curtail and AFRL and was funded entirely by Curtail. No AFRL funding was involved at any stage. The Air Force approved the study for public release with unlimited distribution.
Source: AI-Generated Code Safety and Security, Tables 7.1, 7.5 and 7.6. Distribution (A) Approved for public release; distribution is unlimited; AFRL-2026-3793, AUG 2026
Put the same evidence in front of your team.
17 language models, 6 providers, cleared for public release. We will show you what the same comparison finds in your own traffic.
See the study, then see your own traffic.