Run an overnight eval loop where coding agents fix a reporting agent

Siddharth Deshpande's team let two Claude Opus coding agents test, fix and review a report-answering agent overnight, taking it from 4 in 10 right to 435 of 470 test questions in a week.

Done withClaude· Nov 14, 2026

What they did

Deshpande's team builds a logistics platform for warehouse and yard operators, and one customer wanted to ask plain questions about about 118 reports from a chat client. They wrote two MCP servers on Azure Functions, one with 7 tools to find, run and read reports and one with 19 lookups that turn names like a depot into ids, with a Claude Sonnet 5.5 agent on top. Real questions failed six times in ten. So they wrote 470 graded test questions and built a loop on a cloud machine: deploy to QA through CI, ask every question as real test users, grade each answer, diagnose each failure to one cause, have one Claude Opus coding agent make the smallest fix with a unit test, and have a second Opus agent review the diff without seeing the first one's reasoning. Approved fixes were deployed and failures re-run, up to five rounds a night.

How it went

A 17-question pilot passed 2. After two days of looping the agent passed 435 of 470, or 435 of 441 leaving out cases broken inside QA itself. The loop made 8 commits to server code and 4 rounds of prompt changes, and none had to be reverted. Most fixes were clearer errors and inputs that accept names as well as numbers; nine reports that returned nothing started returning data. The reviewer caught a fix that would have let a gate report run with no site scope. The whole project took one week, and the team's job shrank to deciding what right looks like and reading the morning report.

Worth knowing

Rules were set before the first run: a read-only database user, the email tool disabled, no edits to test questions or graders, no new tools, no touching auth or deploy scripts, commits only to a feature branch and deploys only to QA. Expect a graded agent to try to fix a failing test by changing the test. The post names the models but not the coding agent harness.

Try it

With Claude· 5 parts to fill in

Act as the fixer in an eval loop for [my agent]. Each round, read the failing cases from [eval results file], with their tool-call traces and server logs. For each failure, name one cause: the prompt, [tool server], the harness, or the test. Make the smallest change that fixes it and add a unit test. Never edit the test questions, expected answers or graders, never add new tools, and never touch auth, secrets or deploy config. Commit to [feature branch] only. Then stop and hand the diff to a separate reviewer; after [number] rounds, write a morning report of what passed, what was fixed and what needs a human.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related