Audit 140 background jobs for idempotency and add retry guards

After a retried job double-charged 37 customers, a developer used Claude Code to classify 140 Celery tasks by side effect, roll out idempotency keys and build a run-it-twice test that caught 14 more.

Done with

Claude Code

Apps touched

Postgresql

ReportedOct 15, 2026

What they did

A Celery worker was OOM-killed after charging a card but before acknowledging the message, the broker redelivered it, and 37 customers were charged twice. The developer, yureki_lab on dev.to, then spent two weeks auditing all 140 background jobs in a six-year-old Django and Celery on RabbitMQ codebase. Claude Code first walked every task and filled a fixed YAML schema per task: side effects, reversibility, retry policy, risk level and a file and line range as evidence, marking anything uncertain as unknown rather than none. The developer hand-wrote one guard that claims a key in a Postgres table with a unique constraint, then had Claude Code apply it to every non-safe task in batches of about ten, one PR per batch, critical jobs first, plus a pytest harness that runs each task twice and compares state.

How it went

The inventory sorted the jobs into 9 critical, 23 high, 38 medium, 61 safe and 9 unknown. The developer checked every critical and unknown job and a sample of 20 others; 17 of the 20 held up, and the 3 misses were jobs rated medium that were really high. On its first run the run-twice harness failed 14 tasks the inventory had called fixed or safe. Claude Code wrote about 90 percent of the diff. The harness took it roughly 20 minutes to scaffold and the developer about two hours to tune, and it now runs in CI on every PR that touches a task.

Worth knowing

The agent tracked explicit calls well but kept missing implicit side effects: Django post_save signals, save() overrides and a database trigger. Ask it to list signals and triggers first. The first batch treated an already-processed key as an error, so Celery retried it into the dead-letter queue and flooded alerts. The payment call also needed a provider-side idempotency key built from the business event.

Try it yourself withTry it withClaude Code

4 parts to fill in

Audit every background job in [this repo, e.g. Celery tasks] for what happens if it runs twice. For each task, fill one entry with this schema: task name, side effects (kind, target, reversible), retry policy, risk level (critical, high, medium, safe, unknown) and evidence as file and line range. Follow calls at least [3] levels deep, list Django signals, save() overrides and database triggers explicitly, and mark anything uncertain as unknown, never none. Stop and show me the inventory before changing code. After I approve, apply [my existing idempotency guard] to the non-safe tasks in batches of [10], critical first, treating an already-processed key as success, and add a test that runs each task twice and asserts the state is unchanged.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.