Local speech-to-speech replies in 2.37s, LLM eats 1.36s

The setup is a cascaded speech-to-speech pipeline: separate models for voice activity detection, turn detection, speech recognition, the language model, and text-to-speech, all running on one GPU. A Windows PC supplies only microphone and speaker, connecting over an SSH tunnel, while speech-to-speech itself listens as an OpenAI Realtime-compatible WebSocket server and manages conversation history, turn handling, and interruption.

Breaking the 2.37 seconds apart, VAD took about 0.10 seconds, Smart Turn plus STT about 0.81 seconds, the LLM 1.36 seconds, and TTS time-to-first-audio about 0.10 seconds. The tester notes the logs show the full answer being generated before TTS begins, suggesting the pipeline may be blocking on complete responses — a handoff issue rather than raw model speed. TTS itself looks like a non-bottleneck, generating audio roughly 8 times faster than real time in a single-model test. Smart Turn also proved consequential: it correctly judged a mid-sentence pause as an unfinished utterance and kept the turn intact, and the system canceled in-progress work when the user kept speaking after the AI had started replying. In a separate session with longer history, LLM times ran 1.59 to 2.18 seconds with inputs above 500 tokens, but the tester cautions that the small sample makes a proportional slowdown claim premature.

The stakes may hinge on whether streaming the LLM's output to TTS sentence by sentence delivers the latency drop the tester expects. If it does, local voice assistants on modest single-GPU hardware could look more viable for privacy-sensitive deployments; if not, the remaining gap may push such use cases back toward larger compute. The tester also flags that fully offline operation — with no external communication at all — has not yet been tested, which is likely the benchmark that matters most for environments barred from sending audio out.