Streaming STT cuts AI voice reply to p50 1.594 seconds

The project started as an experiment to have an AI speak in a slow, character-style voice, built from AquesTalk for speech synthesis, whisper.cpp for recognition, and an LLM in between. That version worked, but the pause before a reply became obvious once the interaction was spoken rather than typed. The developer's answer was to rebuild the speech recognition side so text arrives while the user is still talking.

The streaming model is only part of what changed. VAD decides the user has paused, Smart Turn decides whether the turn is really over, and an LLM begins speculatively on the interim text. If the final transcript matches what was guessed, that answer is kept; if not, it is thrown away and regenerated. In the 100-turn test, 74 of 100 replies were kept this way. A separate Whisper path also uncovered a timeout that was cancelling the final recognition step, which the developer fixed by separating committed work from speculative timeouts.

The stakes now hinge on whether these gains survive outside the developer's own machine. The spinning-thread problem only appeared across a long soak test, so the runtime's value may depend on whether it can keep measuring and catching such slowdowns. Software teams building voice features — and anyone making AI characters that need to answer at the right moment — are the ones likely to feel the difference.