Trace and stop an agent copying a stale caveat into sales documents

A developer used their agent's session logs to trace how one evaluation caveat spread into a runbook, a demo guide and a CEO deck, then tested three fixes over five fresh sessions per task.

Done withCustom agent· Nov 20, 2026

What they did

JaviMaligno uses an agent to write manuals, demo runbooks, guides and decks for an industry classification service. They noticed that its sales material kept claiming the service does not classify companies by their name, a caveat that came from a flawed client test set and means nothing to a buyer. The agent keeps full session transcripts, so they traced the sentence read by read: from a reviewer manual and one line of a presenter script into a demo guide's central message and then a slide for the CEO, without anyone asking for it. They then tested three fixes against none: a prohibition in every place sessions load instructions, a role plus a short product story, and that plus a corrected runbook, over four writing tasks with five fresh sessions each.

How it went

With no fix, presenter guides built from the old runbook repeated the claim 5 of 5 times and brochures 3 of 5. The prohibition removed it everywhere, but only for the sentence it named. The role and product story cut the guide case to 3 of 5; correcting the source runbook took it to 0 without naming the sentence anywhere.

Worth knowing

The agent starts from the nearest existing document, so whatever is well documented gets copied forward and compressed into quotable lines. The author argues fixing the source beats banning a sentence. The agent product was not named.

Try it

With Custom agent· 4 parts to fill in

Search your past session logs and [my docs folder] for the sentence or claim [claim that keeps appearing]. For each document that contains it, list which earlier files the session read before writing it and when, so we can see how it spread. Then propose a fix at the source: rewrite [the originating document] from [my product story or brief] and say which documents should be archived. Do not add blanket prohibitions and do not edit anything until I approve the list.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related