Answer buyer questions for an online store and turn every failure into a test

Colakang runs a custom sales agent that answers buyers on their own store, grades its chats weekly and keeps 92 real failures as regression tests; blind scores rose from 3.44 to 4.38 out of 5.

Done withCustom agent· Nov 17, 2026

What they did

The author, Colakang, built a sales agent that replies to buyers on their own online store. It looks up product documents in a vector index and answers questions on listings, prices, stock and fit, offers quotes, routes requests and confirms pickups. Each week the author reads its transcripts, and a judge model grades every conversation from 1 to 5 on accuracy, tone and completeness. Each bad reply becomes a case with the conversation, the wrong sentence, the stage that caused it, the fix and one check that fails on the old version. All 92 checks re-run before any prompt or model change ships. Midway the agent was rebuilt from a drag-and-drop flow builder into code with separate, logged retrieval, drafting and sending stages.

How it went

On the same set of conversations, the rebuild lifted the blind score from 3.44 to 4.07 and cut serious mistakes from 22 to 8. A March run passed 53 of 70 chats. A May audit of 254 chats averaged 4.38 with 220 passing. Failures included quoting a pack price from another product, confirming a Saturday pickup that policy forbids, and twice sending a raw internal string to a buyer. Prices from other products still leaked into retrieval in about 9.5% of sessions after the fixes.

Worth knowing

The scores are one model grading another, not buyer feedback, and the author has no figure linking them to sales. Keeping every product in one shared vector index is what let prices cross between products; a metadata filter or separate stores is the planned fix. The post ends with a pitch for the author's AI-agent company.

Try it

With Custom agent· 5 parts to fill in

You answer buyer questions for [my store] using only the product documents in [my knowledge base]. Quote a price only from the document of the exact product the buyer asked about, and say so if you cannot find it. Follow [my pickup and scheduling policy] and hand off to [a human contact] for anything it does not cover. Separately, each week read [last week's transcripts], score each one 1 to 5 for accuracy, tone and completeness, and for every reply below 4 write a case: the conversation, the wrong sentence, the stage that caused it, a proposed fix and one replay test. Show me the cases and wait for my approval before changing any prompt or setting.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related