Replace manual checks on agent PRs with CI checks that catch accidents

Hiroki Kondo moved three of his hand checks on Claude Code's medium-risk karasu PRs into CI, including a render diff over 3,281 outputs, in a week where 114 agent PRs merged.

Done withClaude CodeinGitHub· Nov 8, 2026

What they did

Kondo writes none of karasu, his architecture modelling language, by hand; Claude Code opens every PR, and each PR gets a risk tier from the paths it touches. On medium-tier PRs he had been checking things himself, and some risky changes were slipping into that tier. He listed those manual checks and turned three into machine checks. Security-boundary paths, parser paths and diffs that add patterns like innerHTML or eval now go to the high tier and need a security review. A test-handling check requires fix PRs to add tests or state why not, flags weakened tests, and fails refactor PRs that touch existing tests. A Render equivalence check renders every example and acceptance file at the merge base and at the PR head and compares the SVG, diagnostics and formatter output. He also had a subagent review the weakened-test check.

How it went

Render equivalence compares 3,281 outputs at about 2 seconds per side; changing one layout constant as a test produced 2,129 differences. Run against past PRs, 10 code-moving refactors passed and 2 that rewrote tests failed. The week closed 117 PRs and merged 114, 27 of them low-tier PRs merged without him reading the diff. He no longer checks medium PRs for unchanged behaviour or test changes, but render equivalence is not yet required while he watches for false positives. A tour bug that lost edits slipped past these checks and was caught by CodeRabbit, then reproduced and fixed by the agent.

Worth knowing

The checks are built only for honest mistakes. The subagent review found 36 ways around the weakened-test check, such as commenting a test out or inserting an early return; he closed the ones you could write by accident and left deliberate disguises to a human reading the diff. Separately, a gh wrapper meant to block premature ready-for-review kept leaking through flags like --repo until it was rewritten to read the target PR itself.

Try it

With Claude Code· 6 parts to fill in

Look at the pull requests you open in [my repo] and the checks I still do by hand on [the medium-risk tier]. List each manual check and say whether a CI job could catch the accidental version of it. For the ones that can, write the jobs: route PRs touching [security-sensitive paths] or adding [risky patterns] to the higher tier, require fix PRs to add tests or give a reason, fail refactor PRs that change existing tests, and compare [rendered outputs or snapshots] between the merge base and the PR head. Try each job against [the last 20 merged PRs] and report false positives. Show me the results before making any job required.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related