Audit feature flag reads against live experiment config

Intuit engineers built CodeSense, an agent that traces how code reads each feature flag and checks it against the flag's live targeting; it reached the right verdict about 95% of the time.

Done withCustom agent· Nov 1, 2026

What they did

Ankur Bhardwaj describes a system his team at Intuit built for a bug that never crashes: code reads a feature flag locally when its targeting rules can only be resolved by the evaluation service, so customers silently get the default. A developer asks in chat or by API to check a flag key in a repo, at a live or draft version. Each request gets a fresh sandbox with the repo checked out, an integration-specific skill describing how that SDK reads flags, and read-only tools for the experimentation platform. The agent finds every read of the key through constants, wrappers and shared libraries, pulls the flag's production and pre-production config, sorts its traits into local-friendly and remote-only, and returns one of seven verdicts with the file, line and a suggested fix. Web plugins were the first integration.

How it went

On the cases the team has run so far, the verdict was correct about 95% of the time. Every miss came from a wrapper library the agent had not seen; after it was given a guide for that library, those cases came out right. Results are advisory only: the agent suggests a diff and a person decides. The author gives no figures yet for production volume; those are promised in later parts of the series.

Worth knowing

The design leans on honest unknowns: the agent may answer UNVERIFIED or UNKNOWN, and if the config cannot be fetched it never falls back to reading the code alone. Each run is sandboxed with no secrets, so a prompt injection in a README has nothing to take. A run that does not report its verdict through an explicit call counts as failed. This is the builders' own account, on a personal blog.

Try it

With Custom agent· 4 parts to fill in

Check how [flag key] is read in [repo] against its live configuration on [my feature flag platform], version [live or draft]. Find every place the key is read, following constants, wrapper functions and shared libraries; do not call it unused until every search path is empty. Fetch the flag's targeting rules and decide whether the client can evaluate it locally or it needs a remote call. Report one verdict (correctly local, correctly remote, incorrectly local, incorrectly remote, not consumed, or unverified with a reason), the file and line, and a suggested fix as a diff. Do not edit the repo. If you cannot fetch the config, say so and stop.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related