Red-team a fine-tuning plan with parallel skeptic agents

Giulio D'Erme had a Claude Code ultracode session attack his Qwen fine-tuning plan; within two hours it found ten blockers, including a training script that would have rewarded every answer with zero.

Done withClaude CodeinOpenRouter· Oct 17, 2026

What they did

D'Erme was planning FT-4, a LoRA and GRPO fine-tune of Qwen3.5-9B meant to use his own memory system, tested at 4 bits on an old 8 GB GTX 1070 Ti. Before paying for training GPUs he asked one Claude Code session, running Opus 5.5 at xhigh effort with ultracode on, to play devil's advocate. Ultracode turns a prompt into a workflow script that fans out subagents: here read-only readers over his records, harness code, memos and papers, ten skeptics told to refute each group of claims, and a critic that checked the review against his brief. He then asked for separate sessions per issue, and the coordinator started eight sessions, wrote their briefs, agreed which session owned which shared file, relayed his answers and handed out rented GPUs from vast.ai. All API spend went through one OpenRouter key he capped at USD 50.

How it went

The first verdict came back in two hours: GO WITH CHANGES, with ten blockers. Skeptics confirmed no GRPO reward code existed, so training would have scored every answer zero, and readers found his untrained harness already inside the range predicted for the trained model, which a later GPU run confirmed. Over about 45 hours it ran four workflows and nine sessions, one of which started a tenth, while he typed roughly 20 turns. The sessions also built a scorer and reward module with 189 new tests and a fix for 4-bit serving. A later planning workflow corrected 26 claims and added 18 missing items.

Worth knowing

He did not measure token spend, so there is no cost figure, and every subagent uses its own context, so a workflow costs far more than one careful turn. The coordinator ran out of context on day two and continued from a summary. Spending approvals did not pass between sessions: one refused to spend on a yes relayed by a peer. He was the only approver for all ten sessions.

Try it

With Claude Code· 3 parts to fill in

ultracode: try to refute [my plan or design doc] before I spend [money or time] on it. Read [these files, notes and papers] read-only. Group the plan's claims and send one skeptic per group whose job is to refute it by default, reading the cited files and full papers rather than abstracts. Ask what would void, confound or weaken the result, and whether a cheaper route reaches the same goal. End with a critic that compares the review against this brief and lists what nobody checked. Return a verdict (go, go with changes, stop), a numbered list of blockers with evidence, and the decisions only I can make. Do not edit any file or spend any money.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related