What they did
The author, Colakang, built a sales agent that replies to buyers on their own online store. It looks up product documents in a vector index and answers questions on listings, prices, stock and fit, offers quotes, routes requests and confirms pickups. Each week the author reads its transcripts, and a judge model grades every conversation from 1 to 5 on accuracy, tone and completeness. Each bad reply becomes a case with the conversation, the wrong sentence, the stage that caused it, the fix and one check that fails on the old version. All 92 checks re-run before any prompt or model change ships. Midway the agent was rebuilt from a drag-and-drop flow builder into code with separate, logged retrieval, drafting and sending stages.
How it went
On the same set of conversations, the rebuild lifted the blind score from 3.44 to 4.07 and cut serious mistakes from 22 to 8. A March run passed 53 of 70 chats. A May audit of 254 chats averaged 4.38 with 220 passing. Failures included quoting a pack price from another product, confirming a Saturday pickup that policy forbids, and twice sending a raw internal string to a buyer. Prices from other products still leaked into retrieval in about 9.5% of sessions after the fixes.
Worth knowing
The scores are one model grading another, not buyer feedback, and the author has no figure linking them to sales. Keeping every product in one shared vector index is what let prices cross between products; a metadata filter or separate stores is the planned fix. The post ends with a pitch for the author's AI-agent company.