Review a sample of chatbot conversations every morning for bugs

Lialia Sakhno builds Calorize, a nutrition app whose assistant, Mira, logs food and edits a diary through tool calls. Every conversation is stored in a self-hosted Langfuse, several thousand turns a day. Instead of reading those traces by hand, she set up a scheduled task in Claude Code that starts an Opus session each morning. A short script pulls the previous day's turns, groups them into one conversation per user per day, and picks 50 at random with no filtering. Claude reads each one in full, tool calls and results included, with access to the project's code, and only reports a finding when it can name the mechanism and the place to fix it: a line of code, a tool description or a prompt section. The full report lands in the Claude Code chat and a short one goes to Telegram every day, even when nothing was found. The task text keeps a section of things already ruled out, which grows each time she says a finding is not a bug.

She says she now catches more bugs than when she read traces herself. Examples it found: a memory tool that kept only the first 200 characters of a fact while telling Mira it was saved, live for seven days; a create-product tool that returned an existing product as newly created, so Mira retried three times and ran out of steps; and lost reasoning between steps that made Mira redo finished work. Replaying one turn 20 times, the redo happened in 5 runs without the reasoning and none with it. None of the fixes were prompt rules. One review is about a half-hour session, and she no longer opens Langfuse unless a finding points to a trace.

The reviewer is read-only: it edits no code or data and messages nobody. Conversations carry an internal number, not names or emails, but the text still goes to a model API. She chose this over Langfuse's built-in LLM judge because a judge needs criteria written in advance, scores turns in isolation and cannot see the code. Abusive messages are kept out of the report by a rule in the task.