Code Generation: Model Architecture
Understand how to select and size base models for code generation within strict latency budgets. Learn context window trade-offs and how retrieval-augmented generation improves suggestion quality while maintaining performance. Gain insight on balancing model parameters, inference latency, language coverage, and security for building responsive, accurate code generation systems.
With a quality-filtered training corpus, a fill-in-the-middle (FIM) objective, and contextual inference signals like open tabs and cursor position already established, the next decision determines whether the system actually ships. The model architecture is where training investments meet production reality. In an ML system design interview, the question lands like this: given a sub-200ms end-to-end latency budget for inline code suggestions, how do you select a base model, size its context window, and augment it with retrieval to maximize suggestion quality?
This is not a theoretical exercise. Production systems like GitHub Copilot, Amazon CodeWhisperer, and Cursor live or die by these choices. A model that produces brilliant completions in 400ms feels sluggish in the IDE, and developers disable it within a day. A model that responds in 80ms but suggests irrelevant code erodes trust just as fast. Staff+ interviewers expect you to quantify these trade-offs, not just name the components.
Base model selection trade-offs
Selecting a base model for code generation involves navigating a three-axis trade-off space. Each axis pulls against the others, and the goal is to find the configuration that maximizes suggestion quality within the latency envelope.
The three axes break down as follows:
Model size (parameter count): Larger models in the 7B–15B range achieve higher
pass@1 The probability that the first generated code sample passes all unit tests in a benchmark like HumanEval, serving as a proxy for single-attempt code correctness. Language coverage: A 3B model trained predominantly on Python and JavaScript may handle those languages well but underperform on Go, Rust, or Kotlin. A 7B general-purpose code model covers more of the long tail, but at higher compute cost.
Inference latency: Smaller models in the 1B–3B range comfortably fit within the sub-200ms budget on a single GPU, while 13B+ models exceed it without aggressive optimization.
The practical sweet spot for production systems sits in the 1B–7B range, using models trained or fine-tuned specifically on code. Techniques like ...