Home AssistantWhat they did
The poster wanted voice control at home without buying expensive smart speakers, and wanted the AI in charge with Home Assistant as its tool rather than the reverse. They bought a Spotpear ESP32-S3 speaker with a round 1.28 inch touch screen and replaced its stock Xiaozhi firmware. An AI coding assistant they call Pip, whose product they do not name, wrote most of the code and did much of the debugging, including finding that the speaker was running out of memory. The poster set the design, made the calls and tested everything on the hardware. The result is speaker firmware with on-device wake words, plus a Docker server running Whisper and Piper that connects to any OpenAI-compatible model (Ollama, OpenAI, DeepSeek, Anthropic or a Hermes agent). It shows up in Home Assistant over MQTT and has an MCP server for announcements and calls.
How it went
It works on that one speaker model, and the poster is the only tester so far. In the demo the poster ran Hermes with a local LLM, and commenters timed answers at around ten seconds, which several called too slow for conversation. The poster said part of the delay was Whisper on a CPU and most of it was the LLM. Both repos are MIT licensed.
Worth knowing
Only the 1.28 inch Spotpear ESP32-S3 BOX is supported; similar-looking 1.85 inch and screenless versions are not. One commenter with a sub-second local setup suggested streaming every stage, moving Whisper off the CPU and using a turn-detection model. Others were wary of handing an AI control of their whole Home Assistant setup.