Vercel AI Gateway adds real-time voice agents, speech-to-text

Key takeaway

• Vercel's AI Gateway now supports real-time voice conversations, text-to-speech, and transcription alongside its existing text and image capabilities.

• Real-time voice agents can hold natural conversations where users interrupt and talk over the model, unlike traditional pipelines that chain separate speech-to-text, language model, and text-to-speech steps.

• All audio calls route through the same unified API gateway with shared observability and spend controls.

3 Key Points

FAQ

Which AI models power the voice features?

Audio capabilities launch with models from OpenAI and xAI. Real-time voice uses OpenAI's gpt-realtime-2, text-to-speech uses xAI's grok-tts, and transcription uses OpenAI's whisper-1.

What makes real-time voice different from speech-to-text chains?

A single real-time model hears audio and produces audio directly. In contrast, a traditional pipeline runs speech-to-text, then a language model, then text-to-speech as separate steps. Real-time also enables users to interrupt and talk over the model (barge-in) with no client-side silence timers.

Is this available now and what SDK is required?

The audio features are in beta and available in AI SDK 7. Developers can also test audio models in a browser-based playground without writing code.