Catch false-green firmware tests with a separate mutation-testing reviewer agent

Omkar Mujumdar was building firmware for an STM32 board that acts as an I2C slave to a Linux host, using apra-fleet, his team's multi-agent setup. A doer agent on a Windows lab PC with the toolchain and ST-LINK wrote the code and a host test suite, and reported mutation-testing its own work. A reviewer agent on another machine, with its own checkout, could build and run anything but never wrote production code. An orchestrator handed out tasks with acceptance criteria written up front. The reviewer's rule was to prove each test by building the bug it claims to catch, re-running mutants the doer said it killed, and running only from a clean build.

The suite showed 274 green tests. Over three review rounds, about a day and mostly unattended, the reviewer found five that could not fail for the stated reason: one checked the error status but not that the I2C bus stayed idle, one took its expected count from the same macro as the code, one tested less than its name said, one could only be proven on the target, and one forbade the correct cache fix. A sixth false green came from stale test binaries that kept passing after the compiler broke. Token cost was not logged.

This is from the vendor's own blog and promotes apra-fleet; the models behind the agents are not named. The stale-binary trap was caused by Git for Windows putting older MinGW DLLs first on PATH, so every task now deletes the build tree before claiming a pass.