Customer case study
Lucihub® & Curtail ReGrade®
Azeem Mohammad at Lucihub
Lucihub now uses Curtail ReGrade to test each release – taking only ~20 minutes. Running ReGrade provides broad testing and caught an API change nobody had documented.
When Lucihub's engineers moved to AI-assisted coding, code shipped faster, and QA's manual impact analysis grew with it. In a four-week pilot, Curtail ReGrade with Claude Code put regression checks on 1,154 real API calls across five user roles, without anyone having to write a single test assertion.
"Since our team moved to AI-assisted coding, every deploy carried more change than QA could check by hand. ReGrade gave us a regression baseline across our API with no test assertions to write, and a release verdict in about 20 minutes, including an API change nobody had documented."
At a glance
Pilot: September 25 to October 20, 2026, on Lucihub's staging environment.
| Metric | Result |
|---|---|
| API requests under regression checks | 1,154 per run, across 5 user roles and 31 endpoints |
| Raw differences explained by the noise profile | 100% of 5,217 |
| False positives on unchanged code | 0 across 6 consecutive replays |
| Release comparison time | about 20 minutes for all 5 roles |
| Regressions in the tested release | 0 |
| Undocumented API changes caught | 1 |
| ReGrade overhead | under 1 ms for 95% of requests |
The challenge
After every deployment, QA ran manual regression cycles to check whether new code had affected existing features. That load grew once engineering moved to AI-assisted coding: more changes shipped more often, and each could touch shared code other features depend on.
- Impact analysis by hand. There was no quick way to see which behavior had actually changed, so QA retested broad areas.
- UI-driven QA can miss API failures. A screen can look right even when the API call behind it fails.
- Five roles, no API test suite. Writing assertions for every endpoint and role wasn't realistic with features changing weekly.
The approach
ReGrade sits as a proxy between the browser and the staging API and records every call.
A replay sends the same requests again and compares each response with the recording.
A profile maps dynamic values and filters expected differences, such as auth tokens and expiring file links. The remaining differences are reviewed to determine whether behavior changed as intended.
Claude Code did most of the hands-on work. It wrote a 10-flow Playwright suite (123 steps, about 9 minutes per role), sorted the differences, and proposed profile rules for a person to approve. Each of the five roles was recorded separately.
Results
- From thousands of differences to a clean signal. The first role replays produced 5,217 raw differences, almost all expected churn. Claude Code grouped the remaining 351 into 8 patterns in one session, confirmed none changed a status code, and proposed narrow rules. After approval, every role replayed with zero unexplained differences.
- No false alarms. Six consecutive replays against unchanged code produced identical results.
- A release verified in about 20 minutes. Against a new staging build, ReGrade found 0 regressions and 25 differences (about 2% of requests), all from one new field missing from the release notes. Follow-up exposed a design gap: the field showed 0 instead of "unknown" for older records. The fix was in progress at the time of the pilot.
- Hidden issues surfaced. Recording real traffic exposed silent API failures and places where the UI and API disagreed. Manual QA had missed some of them because the screens looked fine.
- AI that knows when to stop. Claude Code refused a rule that would have broken replays, kept credentials out of its summaries, and asked before changing the shared profile.
Why it matters for teams coding with AI
- An independent check. Tests written by the same assistant that wrote a feature can share its blind spots. A recording of real user behavior provides a separate reference for comparison.
- It catches what nobody announced. The replay flagged an undocumented change within minutes of the deploy.
- Impact analysis becomes a short list. QA can focus its review on the endpoints and roles where behavior changed, then determine which changes were intended.
- No assertions to maintain. Once a release is accepted, the baseline is recorded again in about 45 minutes for all five roles.
Implementation considerations
- API coverage. ReGrade checks API behavior. UI-only regressions need separate checks.
- Recorded coverage. Record enough traffic to exercise the application, add flows for new features, and have a person decide which changes were intended.
- Human review. An API difference is not necessarily an error. AI coding agents can help assess which differences are likely unintended.
- Write flows need care. Replays repeat recorded requests and can create or delete data. Account for those effects before replaying traffic against an application. This pilot used read-only recordings. Lucihub plans to expand future test coverage to all functionality.
- Profile setup. Each customer needs a profile to map dynamic values and filter noise. The customer works with their AI agent to build it before replaying traffic, typically as a one-time setup.
Best fit: API-backed web apps with several user roles, frequent deploys (especially with AI-assisted development), and a stable staging environment.
Areas for improvement
- Setup friction. Certificate handling, profile import, and expiring-URL rules required workarounds. Lucihub shared this feedback with Curtail.
What's next
Lucihub now uses the five-role release comparison for every release. Next, the team plans to measure QA time before and after, plant deliberate API changes to measure the detection rate, and run replays in CI once that's supported.