Amazon SageMaker AI and vLLM enable real-time speech-to-text via bidirectional streaming, starting November 2025

Starting November 2025, Amazon SageMaker AI supports bidirectional streaming for real-time inference, allowing data to flow continuously in both directions between clients and model containers over a single persistent connection. vLLM's Realtime API uses WebSockets to enable real-time audio transcription.

The solution deploys Voxtral-Mini-4B-Realtime-2602 (Mistral AI's compact real-time speech model) to a SageMaker AI endpoint using a vLLM container with bidirectional streaming. SageMaker AI automatically bridges HTTP/2 event streams on the client side with WebSocket connections to the container, eliminating the need to build custom protocol translation layers.

Voice AI applications receive transcription tokens as audio arrives rather than waiting for a complete recording, eliminating latency that breaks real-time experiences. vLLM applies piecewise CUDA graph execution to reduce GPU kernel launch overhead and lower per-token latency during streaming transcription.