Which AI models power the voice features?

Audio capabilities launch with models from OpenAI and xAI. Real-time voice uses OpenAI's gpt-realtime-2, text-to-speech uses xAI's grok-tts, and transcription uses OpenAI's whisper-1.

What makes real-time voice different from speech-to-text chains?

A single real-time model hears audio and produces audio directly. In contrast, a traditional pipeline runs speech-to-text, then a language model, then text-to-speech as separate steps. Real-time also enables users to interrupt and talk over the model (barge-in) with no client-side silence timers.

Is this available now and what SDK is required?

The audio features are in beta and available in AI SDK 7. Developers can also test audio models in a browser-based playground without writing code.

Audio & SpeechVercel AI BlogJun 29, 2026Jun 30, 2026, 01:00 JST

Vercel AI Gateway adds real-time voice agents, speech-to-text

Send on LINE

3 Key Points

What happened
Vercel's AI Gateway now supports audio and voice capabilities including real-time voice conversations, text-to-speech, and speech-to-text transcription. These features launch with models from OpenAI and xAI, available in beta through AI SDK 7.
Why it matters
Real-time voice agents let applications hold live conversations where users can interrupt and talk over the model as they would with a person, making it practical for voice assistants, customer support, and hands-free tools. Unlike chaining separate speech-to-text, language model, and text-to-speech steps, a single real-time model hears and produces audio directly.
What to watch
Audio calls route through the same AI Gateway infrastructure as text and image requests, meaning developers can manage all modalities—voice, text, video—with one API key, unified observability, and shared spend controls. A browser-based playground lets developers test audio models without writing code.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

FAQ

Which AI models power the voice features?: Audio capabilities launch with models from OpenAI and xAI. Real-time voice uses OpenAI's gpt-realtime-2, text-to-speech uses xAI's grok-tts, and transcription uses OpenAI's whisper-1.
What makes real-time voice different from speech-to-text chains?: A single real-time model hears audio and produces audio directly. In contrast, a traditional pipeline runs speech-to-text, then a language model, then text-to-speech as separate steps. Real-time also enables users to interrupt and talk over the model (barge-in) with no client-side silence timers.
Is this available now and what SDK is required?: The audio features are in beta and available in AI SDK 7. Developers can also test audio models in a browser-based playground without writing code.

Vercel AI BlogRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.