Model and Context Trade-offs

A design that works well in a pilot with fifty users can become too slow or too expensive when it serves fifty thousand. Most of that difference comes from two decisions: which model handles each request, and how much context is sent with it. Every token sent to or generated by a model adds cost and latency, and those small amounts multiply quickly at enterprise scale. A customer’s finance team will eventually ask what each conversation costs, and an FDE needs a clear, defensible answer.

FDE interviewers often raise these trade-offs as follow-up questions in system design rounds. They may ask how to reduce latency, whether the full document set should simply go into the context window, or what the system will cost per month. Strong answers connect each choice to the task’s quality requirements and back it with a simple estimate. This lesson covers model routing, long context versus retrieval, context caching, and cost estimation.

Model routing

Model routing sends each request to the least expensive model that can handle it well. Most enterprise traffic is routine: classifying a request, extracting fields, or answering a common question. A smaller fraction requires deeper reasoning, such as analyzing a complex dispute or planning a multi-step resolution. Sending every request to the most capable model wastes money and adds latency, while sending every request to the fastest model sacrifices quality on the hard cases.

There are several ways to route requests. Rule-based routing uses known signals, such as request type or customer tier, to choose the model. Classifier-based routing uses a fast model to judge the request’s complexity before choosing. Escalation routing tries the faster ...