OpenAI Drops Voice Intelligence API: Here’s What It Changes
Before we dive into this week's topic, one of our hands-on courses — Building with OpenAI: From APIs to Agents — recently got a fresh update. It covers the foundations everything in this issue is built on: working directly with OpenAI's APIs across text and audio, wiring up function calling and web search as tools, and assembling full agentic systems with Agent Builder and ChatKit — all in an in-browser editor with zero setup. If you want those fundamentals in place before you build a voice agent of your own, it's one of the fastest ways to get there.
Suppose you run a customer support operation. Your agents primarily speak English, but many callers use other languages. In a traditional setup, you route those calls to specialized teams, hire multilingual staff, or place callers on hold while locating an available translator. As call volume increases, translation and staffing costs rise.
With live translation, the model can translate speech in near real time, including partial utterances, as the conversation continues. The caller does not need to wait on hold, be routed to another queue, or speak with a second agent.
That's not a hypothetical anymore. OpenAI just shipped two new real-time audio models to the API: GPT Realtime Translate and GPT Realtime 2. And the use cases go well beyond customer support. Here's what they actually do, how they work, and what changes for developers building on top of them.
Why has voice AI kept failing?#
Before getting into what OpenAI shipped, it's worth understanding why voice AI has underperformed for so long despite constant promises.
The problem was always the pipeline. Every voice product you've used in the last decade was built on the same basic architecture: capture audio, run it through a speech-to-text model, feed the transcript to a language model, get text back, pipe that through a text-to-speech model, and play the audio output. Five steps, five potential failure points, five sources of latency stacking on top of each other.
Each handoff added a delay. Each delay made the interaction feel unnatural. And because the language model was operating on text, not audio, it had no access to the paralinguistic information baked into voice: tone, hesitation, emphasis, the half-second pause before someone changes their mind.
This is why Siri feels like a search bar you talk to. It's why Alexa is good at timers and bad at anything else. The architecture treated voice as a transport mechanism for text rather than as a medium with its own properties.
