Search⌘ K
AI Features
Home
Courses
Grokking the Generative AI System Design Interview

Design a Real-Time Voice Assistant

Design a real-time voice assistant that streams audio continuously, handles turn-taking and interruptions, and recovers from failures while keeping latency low. Size it for peak concurrent sessions, and build it from incremental, modular components that balance responsiveness, accuracy, and privacy.

We'll cover the following...

A voice assistant can answer correctly and still feel broken. The failure often happens in the silence after the user stops speaking. If that gap stretches past roughly a second, users start talking over the system, repeat themselves, or assume the call dropped.

This lesson treats that gap as a system design problem. A production voice assistant is not just text chat with audio attached. Capacity is shaped by long-lived concurrent sessions, continuous audio streaming, interruption handling, and strict p95 latency targets. The system must keep listening while it speaks, cancel work mid-response, and recover when audio quality collapses.

We will scope the design around three forces only.

  • Latency governs every stage on the critical path.

  • Accuracy must hold under noise, accents, packet loss, and imperfect endpoint detection.

  • Privacy changes retention, routing, consent, and even whether raw audio is stored at all.

Note: In voice systems, concurrency matters more than requests per second because each session occupies transport, state, and model capacity for minutes.

That framing also sets up the next lesson. Once many real-time models and policies coexist, routing and observability stop being app concerns and become gateway concerns. First, though, we need the voice architecture itself.

System requirements

Requirements decide the design, so fix them before sizing anything.

Functional requirements

  • Turn-taking: The assistant listens while it speaks, detects end of speech, and yields to real interruptions.

  • Streaming dialogue: It transcribes, reasons, and speaks incrementally instead of waiting for each stage to finish.

  • Tool use: It can call tools mid-conversation and speak the result.

  • Consent and retention: It honors recording consent, retention limits, and deletion requests.

  • Failure recovery: It survives packet loss, low ASR confidence, and TTS failure without dropping the call.

Nonfunctional requirements

  • Latency: Silence after the user stops speaking stays under about 1 second at p95.

  • Scalability: The fleet supports 20,000 concurrent sessions at peak.

  • Availability: Regional media edges and failover keep calls alive.

  • Accuracy: Speech recognition holds up under noise, accents, and packet loss.

  • Privacy: Raw audio is encrypted, consent-gated, and kept only as long as policy allows.

  • Cost: Cancellation-aware serving and warm pools avoid paying for abandoned replies.

Resource estimation

Let's size the system from one workload: 20,000 peak concurrent sessions, 4-minute calls, and 50 percent average utilization. Every number below derives from these assumptions.

Storage estimation

Let's start with what persists after a call and what must exist during a call. Voice systems keep hot session state in memory, then move artifacts to colder storage. The main buckets are session metadata, transcripts, optional recordings, evaluation labels, and model assets.

With 4-minute calls and 50 percent average utilization, the fleet averages 10,000 concurrent sessions, or about 3.6 million sessions per day, and only 10 percent are retained with consent. Mono compressed audio runs at 24 kbps per direction, so a retained call holds ...