Search⌘ K
AI Features
Home
Courses
Grokking the Generative AI System Design Interview

Design an Automatic Speech Recognition System

Design an ASR system around a Whisper-style model that preprocesses audio into spectrograms, transcribes it on GPUs, and post-processes the text with confidence-based quality control.

We'll cover the following...

Automatic speech recognition (ASR) systems convert spoken language into text, enabling seamless human-computer interaction. ASR technology is used in virtual assistants, transcription pipelines, voice search systems, and accessibility tools. Recent advancements in deep learning and neural networks have significantly improved the accuracy and efficiency of ASR systems, making them integral to modern applications.

An abstract of an automatic speech recognition system
An abstract of an automatic speech recognition system

Traditional ASR systemsBuilt on hand-crafted features like Mel-Frequency Cepstral Coefficients (MFCCs) and statistical models such as Gaussian Mixture Models (GMMs) and Hidden Markov Models (HMMs). Acoustic, pronunciation, and language models are trained separately. face several challenges, including sensitivity to background noise, speaker accents, and variations in speech patterns, which reduce accuracy in real-world conditions. These systems often have limitations in multilingual recognition, code-switching, and the recognition of diverse accents, which can reduce accessibility and impact inclusivity. Modern ASR models like OpenAI’s Whisper v3 and Google DeepMind’s Canary have significantly improved ASR by leveraging large-scale self-supervised learning on diverse, multilingual datasets. These models demonstrate superior robustness to noise, accents, and low-resource languagesLanguages that lack large datasets, standardized transcripts, or computational tools, making model training challenging., enabling more accurate transcription in real-world scenarios.Built on hand-crafted features like Mel-Frequency Cepstral Coefficients (MFCCs) and statistical models such as Gaussian Mixture Models (GMMs) and Hidden Markov Models (HMMs). Acoustic, pronunciation, and language models are trained separately.

Note: In this lesson, we will look at the fundamental concepts of ASR, the architecture of ASR models, the training process, and evaluation methods to build an end-to-end ASR system. As with any system, we will start with the requirements.

Requirements

The development of an image captioning system begins with identifying the functional requirements that shape its essential behaviors and the non-functional requirements that determine its performance and reliability.

Functional requirements

The functional requirements of an ASR system are:

  • Speech-to-text conversion: The model should accurately transcribe spoken language into text while accounting for pronunciation, accent, and background noise variations.

  • Audio preprocessing: The system must handle noise reduction, echo cancellation, and feature extraction.

  • Real-time processing: The ASR model must support low-latency, real-time transcription for use cases such as live captioning and virtual assistants.

  • Multilingual support: The ASR system must support recognition across multiple languages and dialects while maintaining consistent accuracy.

Nonfunctional requirements

The nonfunctional requirements for an ASR system are:

  • Accuracy: The error rate (e.g., WER) should be minimized, particularly in noisy environments and for diverse accents and dialects.

  • Low latency: The ASR system should process and transcribe speech with minimal delay to support real-time applications.

  • Security and privacy: All user data (audio, text, and metadata) must be securely stored, encrypted, and handled according to data protection regulations.

  • Scalability: The system should handle large-scale deployments, supporting multiple users and languages simultaneously.

With our requirements decided, we can choose a model for our system.

Model selection

Selecting the right model architecture is a key step in designing an automatic speech recognition (ASR) system. Model selection directly affects how well the system can generalize, scale to large datasets, and maintain accuracy under different workload conditions, and impacts both training complexity and deployment constraints.

We will focus on modern self-supervised learning (SSL) architectures for ASR, as they have demonstrated superior generalization and robustness over traditional methods. One such model is Whisper, a transformer-based ASR model that leverages large-scale pretraining on diverse audio datasets to achieve state-of-the-art accuracy across multiple languages and acoustic environments.

The Whisper v3 architecture
The Whisper v3 architecture

The Whisper model follows a sequence-to-sequence transformer architecture trained on a vast dataset of audio-text pairs. The core components of Whisper’s architecture are:

  1. Feature extraction: The input audio is first converted into a log-mel spectrogramA representation that converts audio into a visual format by mapping frequency components over time. The mel scale aligns frequencies with human auditory perception, and logarithmic scaling emphasizes lower-amplitude signals. representation. This is done using a 2D convolutional layer with GELUGELU (Gaussian Error Linear Unit) is an activation function that applies a smooth, non-linear transformation to input values based on the Gaussian cumulative distribution function (CDF) and often improves performance in deep learning models compared to ReLU. activation, followed by adding sinusoidal positional encodingA method used in transformer models to encode positional information in input sequences by adding sinusoidal functions with varying frequencies to each token embedding. to retain the temporal structure of the audio sequence.

  2. Encoder: he encoder consists of multiple encoding layers (blocks) that operate on log-mel spectrogram features. This stage enables the model to learn high-dimensional representations of the speech signal.

  3. Decoder: The decoder takes the encoded speech representation and, using cross-attention and learned positional encoding, predicts text tokens one by one in an autoregressive manner, similar to how large language models like GPT generate text.

In the illustration above, “tokens in multitask training format” refers to how the model is trained to perform multiple speech-related tasks using a unified token-based representation.

Each input sequence starts with special tokens that guide the model on what task to perform. For example:

  • SOT (Start of transcript): This usually marks the beginning of a transcription task.

  • ENEnglish (Language token): We can specify the language of the speech (e.g., English).

  • Other tokens (like timestamps or punctuations): We can provide additional context for tasks such as translation, transcription, speaker diarizationIn speech recognition, diarization is a process of automatically partitioning an audio recording into segments that correspond to different speakers., or summarization.

This multitask tokenization helps the model generalize across different ASR-related tasks, making it more versatile and efficient in handling diverse speech-processing needs.

This system uses a Whisper-like model with 1 billion parameters. With the model architecture defined, the next step is to examine the training process and data requirements needed to train the ASR system.

The training process

Start with how to train a speech recognition model similar to Whisper v3, beginning with the dataset.

Data acquisition and preparation

High-quality, well-curated data is crucial for training an effective ASR model. The dataset must include a wide variety of audio-transcription pairs to ensure the model can accurately recognize and transcribe speech across different conditions. These pairs should cover diverse linguistic features such as different speech styles, accents, tones, emotions, and intonations. The dataset must cover a wide range of contexts and environments to ensure the model can handle diverse acoustic conditions and maintain transcription accuracy in production settings.

Source of training data

We can either build our own dataset or use a publicly available one, such as the Common Voice corpus (v19.0)https://commonvoice.mozilla.org/en/datasets. Collecting audio data from scratch is particularly challenging due to the time and resources involved. Common Voice, for example, contains 3,587 hours of audio; recording that amount alone would take a single person roughly half a year, and that’s before accounting for linguistic diversity and tonal variation.

For our ASR system, we will use Common Voice datasets totaling approximately 50,000 hours of audio, which translates to around 1–2 TB of storage (we’ll assume 2 TB). Based on Common Voice’s data distribution, this ...