Code Generation: Serving & Trade-Offs

With offline benchmarks, acceptance-rate metrics, retention-based A/B tests, and privacy mitigations already in place, one critical design challenge remains. The model must serve completions fast enough to feel invisible inside a developer’s typing flow. A suggestion that arrives 500 ms after a keystroke disrupts the very workflow it aims to accelerate. This is the core interview question you should expect: how do you deliver code completions in under 200 ms while maintaining quality, handling cancellation, and collecting signal for continuous improvement?

Production systems like GitHub Copilot and Amazon CodeWhisperer navigate exactly this latency-quality-throughput triangle every day. This lesson covers the three pillars that close the code generation case study. First, speculative decoding slashes autoregressive latency. Second, a streaming architecture bridges the ML backend and the IDE for seamless UX. Third, a feedback loop converts user behavior into fine-tuning signal. Interviewers at L5 and above expect candidates to articulate concrete trade-offs across all three pillars, not just name the techniques.

Speculative decoding for sub-200 ms latency

Standard autoregressive decoding generates one token per forward pass through the target model. For a 20-token suggestion, that means 20 sequential forward passes. At small batch sizes, GPU compute is underutilized and memory bandwidththe rate at which data can be read from or written to GPU memory, which becomes the dominant bottleneck when the model weights must be loaded for each forward pass rather than being reused across a large batch dominates wall-clock time. Latency scales linearly with sequence length, making vanilla decoding too slow for inline suggestions that must appear within 200 ms.

How speculative decoding works

Speculative decoding breaks this bottleneck by splitting generation into two stages. A lightweight draft model, often a distilled variant with roughly 150M parameters, proposes K candidate tokens autoregressively. Because the draft model is small, each of its forward passes is fast. The full target model then verifies all K tokens in a single parallel forward pass, accepting a prefix of correct tokens and resampling from the ...