Audit a home email triage agent against hand-labelled mail

Federico built his own mail agent, called Foyer, on a Mac mini at home. For a year it read his Gmail and, for each message, set a type and an urgency and decided whether to draft a reply, create a task or ignore it, showing the results on a dashboard. When he set out to write a paper on it, he labelled 231 emails by hand over two afternoons and then 500 more, and used those labels to score the agent and nine other approaches: an old TF-IDF and logistic regression classifier trained on his labels, typed decision models, and local generative models such as Qwen and Gemma.

The type field on the dashboard was right 28% of the time, worse than calling every email an update (58%). The cause was a thirty-line keyword fallback left in place after the local model timed out, while the README still described the model. The small classifier trained on his labels was the most accurate on every field. No model could tell from the text who a sender was to him (52%), but copying the relationship from earlier mail by the same address was right 98.6% of the time, so the agent now looks it up. The most accurate system still ignored 18% of mail that needed action, against 4% for Gemma, until he set a threshold based on what a missed email costs him.

It is one person and one inbox in two languages, and the author says so. A blind second labeller agreed with him on type but barely on whether a mail needs a reply, which is a personal habit, not something in the text. His first export also held a VPN password in clear text, so redact before you label. The full study is on arXiv.