Turn coding agent mistakes into hooks and regression tests

Piotr uses a Claude Code session as an all-day assistant, wired to Slack, mail, a calendar and TaskWarrior, with 86 hooks, 50 skills and 231 scripts. Tired of the agent apologising and then repeating the same error, in October 2026 he built a loop around it. A cron job cuts the transcript since the last run every two hours and sends it to a judge from another model family, run through the Codex CLI in a read-only sandbox. The judge must quote the transcript for every strike or praise and suggests up to three fixes. A reflect subagent then picks a fix, writes a regression test built from the quoted strike, commits it locally and records it in an append-only ledger in git. Fixes must be mechanisms (a context-injecting hook, an enforcing hook, a script or a small-model check), not new lines of instruction text. Anything touching permissions, spending or external writes goes to his board instead.

Across two agent sessions the judge logged 299 strikes. 174 were closed with a committed fix: 92 changed a tool, 52 the harness, and 29 were text edits made before he stopped accepting text. Of those 174, 36 stayed quiet for a day, 107 were too new to tell, and 31 saw the same family of mistake return. One example: after the agent kept answering questions that a form or announcement should have answered, a hook now asks it to propose that edit, and the next day brought three praises and no repeat. He calls none of it proven yet.

A loop that edits its own harness can quietly weaken its guardrails or game its judge. He pins the rubric to a frozen day of 26 of his own complaints, treats lines from email or Slack as untrusted, and counts only text he typed himself. His real metric is his own complaints per week, not the strike count. An earlier attempt that only added rules to CLAUDE.md failed: the agent repeated the mistake while the test that the rule existed kept passing.