What they did
Trail of Bits engineers had written hundreds of security patches for open-source projects with Codex as part of Patch the Planet, a joint effort with OpenAI. After finding that some of those patches broke things, they went back over every patch a Trail of Bits engineer had authored and a maintainer had merged, more than 130 pull requests. For each one an agent traced the changed code out to its callers and real interfaces, listed behaviour that might have broken, and tried to prove each suspicion by running a proof of concept against a built, running copy of the project before and after the change. An engineer then reviewed what it found. The team turned what they learned into two plugins, regression-hunt and patching-bugs, which their engineers now use in the same work.
How it went
One run produced 58 candidate regressions. Engineers confirmed 21 of them, about 36% precision: 16 functional, one performance and four build or test breaks. None of them undid a security protection. The authors call the review noisy, and each new run turned up a different set of candidates to sort by hand. On those 21 known regressions, the regression-hunt plugin gave 25% fewer false positives than plain Codex told to review the PR for regressions, but its stricter evidence rules left some real regressions marked borderline instead of confirmed.
Worth knowing
Everything rests on a live proof of concept. If the agent cannot reproduce a claim on a running instance, the result counts as inconclusive, so you need a project that builds and runs locally. Results get better when the repo holds a written threat model and patching guidelines that say what counts as a regression. This is a vendor post that also launches the plugins, and the numbers are the authors' own preliminary evaluation.