Voice Agents With the Realtime API
Evaluate speech-to-speech and chained pipeline architectures for real-time API voice agents, balancing natural interaction, transcript accuracy, and latency. Configure voice activity detection (VAD) to manage turn-taking and interruptions. Maintain operator permissions, protect data integrity, and design for future workflow enhancements based on user needs.
In Northstar’s first voice rehearsal, an operator said, “What is the dispatch window for NS-318…”, paused to check a note, and kept going: “Actually, that ticket number was wrong.” By then the agent had already decided the turn was over and started a read for NS-318.
A text agent knows when the user is done, because they press Enter. A voice agent has to guess from a pause, and cope when the user starts talking again halfway through its answer. The operator now wants to ask for the NS-318 dispatch window out loud. The authorized source is still dispatch/NS-318/v3, the window is still October 29, 08:00–12:00 UTC, and the agent still only prepares drafts for operator review. Voice changes the interface, not those rules.
Speech-to-speech or a chained pipeline?
OpenAI’s voice agents guide describes three ways to build a voice agent, and the choice shapes everything after it. In speech-to-speech, a Realtime model takes input audio and produces audio directly. Nothing sits between the audio and the reasoning, so turn-taking is natural and fast. Input transcription is optional and runs separately, which makes it a guide to what was said, not an exact record of what the model heard. In a chained pipeline, the audio is transcribed, the text goes to a reasoning model, and the reply is synthesized. The application can inspect the transcript and control each stage, at the cost of added delay at every hop, and a misheard ticket number flows straight into reasoning and speech. The third option, GPT-Live (gpt-live-1), listens and speaks at the same time and hands reasoning and tool use to a separate backend, such as an existing text agent, so the user can keep talking while that work runs. This lesson compares the first two for Northstar’s console.
Northstar’s 20-turn rehearsal showed what the first choice costs. Five responses began before the operator finished speaking, and in two of those five, a read was already in flight when the operator corrected the request. No ticket was written and no customer message sent.
For an interactive console, start with a Realtime rehearsal limited to reads and drafts. If Northstar later needs an inspectable transcript at every decision point, compare a chained pipeline on representative audio and count its transcription errors. In neither design does a transcript count as approval. Weigh the two options for the console, then change the weights as if a full transcript became mandatory.
The choice is reversible as long as the business boundary stays fixed. Keep the same ticket authorization, source checks, and operator review in both designs, so a measured comparison changes only the audio architecture.
Where does the credential live?
The operator’s interface connects to an authenticated Northstar service. That service holds the platform key, manages the Realtime connection, and calls a server-side read adapter with the operator’s trusted identity and entitlements. The ...