How AI Reshaped System Design and Interviews
Explore how AI changes traditional System Design assumptions. Learn to balance cost, latency, and quality when integrating large models. Manage request budgets, scaling metrics, and grounding mechanisms. Distinguish deterministic from probabilistic components while addressing safety, relevance, and other nonfunctional requirements in System Design interviews.
In a System Design interview, a candidate may be asked to design a customer support chat system. The candidate proposes an architecture with a client, API gateway, cache, stateless application servers, message queue, and sharded database. This architecture covers the core application infrastructure but does not yet address the requirements of LLM-powered features.
The interviewer then asks how the system will support semantic retrieval, how retrieved company documents will provide evidence for generated responses, how model inference latency fits within the end-to-end response-time budget, and how the system evaluates whether generated claims are supported by retrieved evidence. These requirements introduce additional design considerations for retrieval, inference, latency, and response validation.
Traditional System Design interviews emphasized scalable data storage, efficient data processing, and predictable system behavior. Common architectures used load balancers, caches, sharded databases, and message queues to meet latency, throughput, and availability requirements. These components remain essential, but AI-enabled systems introduce additional architectural requirements. Designing these systems also requires accounting for model inference, retrieval, output validation, and their effects on latency, cost, and reliability.
The additional component is a model inference call. The request path may now include inference from a large language model (LLM) or another machine learning model. A model endpoint may look like a conventional API dependency, but it has different latency, cost, and output characteristics. Model inference can introduce substantial compute costs and latency, while generated outputs may vary between requests. Identical inputs may produce different outputs depending on sampling parameters, model configuration, and execution conditions. These characteristics require additional consideration when designing for cost, latency, caching, and failure recovery.
AI-enabled systems retain traditional architectural components but introduce model inference with additional operational requirements. Model inference introduces compute costs, latency constraints, and potential output variability. These characteristics create trade-offs among cost, latency, and output quality. Improving one may require compromising another. In AI System Design interviews, explain how architectural decisions affect these trade-offs.
Focus on system architecture and design decisions rather than model internals. Examine how AI changes architectural requirements and System Design interview expectations. The next lesson explores how expectations vary across companies and engineering roles.
The new boxes in modern architectures
AI-enabled applications use traditional infrastructure components but may introduce additional components into the critical request path. Each component handles a specific responsibility in request processing.
Review the key components before tracing a request from user input to generated response:
Model inference service: This service runs the model or calls an external provider and returns generated tokens. It acts like an online dependency, but unlike a normal microservice, its latency, reliability, and per-request cost can vary with prompt size and output length.
Embedding pipeline: This offline or nearline pipeline converts text into vectors that capture semantic meaning. An
embedding A dense numeric representation of content used for similarity search Vector database: This store indexes embeddings for nearest-neighbor lookup. It complements transactional databases rather than replacing them, much like a library's subject catalog complements the bookshelf location system.
Retrieval and RAG layer:
Retrieval-augmented generation A pattern where the system fetches relevant documents before generation to ground the model in trusted data inference The process of running a trained model on new input to produce an output, such as generating a response to a user's prompt GPU or accelerator fleet: These machines run inference efficiently, but they are scarce and capacity-sensitive. Throughput depends on batching, memory pressure, and queueing behavior, not only replica count.
Context and memory manager: This layer selects conversation history, trims irrelevant turns, and summarizes older context to stay within the model's window.
Agent orchestration: This controller manages multistep tool use, retries, branching, and stop conditions.
Guardrails and evaluation: These layers check safety, policy compliance, and quality signals inline or shortly after response generation.
The side-by-side view below highlights the architectural components that conventional designs often omit before examining which conventional assumptions no longer hold:
That architectural vocabulary sets up the harder part, which is understanding why the old assumptions no longer hold.
Note: Each new box adds its own scaling and failure modes, so naming the component is never enough in an interview.
Where classic assumptions break
The interview signal changes when you can explain not just the new boxes, but what they do to the old mental model. A useful pattern is to state the old assumption, show the AI disruption, and then derive the new design consequence.
Compute and latency: Traditional online services often rely on comparatively inexpensive CPU, memory, storage, and network resources per request. AI inference can introduce a more expensive compute path, often through GPUs or hosted model APIs. For hosted LLM APIs, request cost typically depends on token usage and model selection.
Compute no longer stays cheap: A simple endpoint once spent money on CPU cycles, storage, and network transfer. Now the same endpoint may spend most of its budget on generated and consumed tokens. That pushes model routing, semantic caching, and context trimming into the architecture itself.
Latency no longer stays tight: A database read may complete in milliseconds, while inference often takes hundreds of milliseconds or several seconds. The system responds by streaming partial output, batching backend work where possible, and separating first-token latency from full-response latency.
Note: Adding a larger model can improve answer quality and still fail the design if the latency budget or daily cost ceiling collapses.
Determinism, quality, and failure: Deterministic services usually return the same output for the same input. A probabilistic model can vary because prompts include timestamps, conversation history, retrieved context, and nonzero decoding randomness.
Correctness becomes a moving target: This weakens exact response caching and makes reproducibility harder. The design often caches embeddings, retrieval results, reranking outputs, or prompt templates instead of caching only final answers.
Availability is no longer enough: The system can be healthy from an infrastructure view and still be wrong, stale, unsafe, or poorly grounded. Nonfunctional requirements now include relevance, hallucination rate, factuality, and safety alongside p95 latency and uptime.
Data shape changes under the query layer: Structured rows and exact-match indexes still matter, but many questions now hit unstructured corpora through semantic similarity.
Reranking A second-stage model that reorders retrieved documents to improve relevance before generation
The comparison below compresses these broken assumptions into a format you can quickly recall during an interview:
AI System Design Shifts
Area | Old Assumption | AI Change | System Design Consequence |
Compute | Capacity is mostly fixed per request | Cost varies by model, context length, and tool use | Use model routing and budget-aware orchestration |
Latency | Response waits for full processing to finish | Users can benefit from partial output as it is generated | Stream tokens/results and design for progressive rendering |
Determinism | Same input should produce the same output | Outputs may vary across runs and model versions | Add guardrails, versioning, and retry/verification paths |
Quality metrics | Correctness is measured by exact rules or test cases | Quality is often probabilistic and task-dependent | Use eval harnesses, human review, and task-specific benchmarks |
Data model | Structured records are the primary source of truth | Useful context may be unstructured, embedded, or retrieved dynamically | Support hybrid retrieval across SQL, vector, and document stores |
Failure modes | Errors are usually explicit and easy to detect | Models can be wrong but confident and fluent | Add safe fallback responses, confidence checks, and escalation paths |
Once the assumptions break, the design conversation shifts from diagnosis to control levers.
The new design levers
AI-enabled systems do not require replacing deterministic components with model calls. A typical architecture combines deterministic services with probabilistic components such as model inference. The key design work is defining where model behavior is allowed, constrained, validated, or bypassed.
Start with the request budget. Every request spends latency, tokens, and money. The system stays feasible only if those budgets are actively controlled.
Cost control: The request first checks the semantic cache, then may route simple prompts to a small model and reserve a larger model for harder cases. Prompt compression and context trimming reduce token load before inference begins.
Scaling metrics: Capacity planning now tracks tokens per second, concurrency per GPU, batch size, and
KV cache Stored attention state reused during autoregressive generation to avoid recomputing prior tokens Grounding and policy control: Retrieval pulls trusted context, citations attach evidence, and moderation or policy filters screen both prompts and outputs. Evaluation gates behave like CI for quality regressions by blocking rollout when answer quality drops.
State management: Short-term chat history stays in the active context window, while long-term memory is summarized or stored externally. Multistep agents often checkpoint workflow state so the job can resume after tool failure or timeout.
Practical tip: In an interview, say which part must remain deterministic. Access control, pricing rules, and final side effects rarely belong inside generation.
The trade-offs are mechanical. Larger context windows may improve grounding but raise latency and spend. Aggressive trimming lowers cost but can remove the very facts the model needs. Strong candidates talk through those levers numerically, not poetically.
With the technical levers in place, we can now connect them directly to modern interview expectations.
How the interview changed
System Design interviews increasingly include AI coding assistants, enterprise knowledge systems that use internal data to ground responses, and shared model inference platforms. Existing System Design problems may also require AI-specific components. Search systems may combine lexical and semantic retrieval, followed by reranking to improve result relevance. Customer support systems may generate responses using relevant information retrieved from ticket histories, knowledge bases, and policy documents. Recommendation systems may run model inference within the online request path to generate or rank recommendations.
A strong answer starts by separating deterministic responsibilities from probabilistic ones. The candidate defines where retrieval happens, what the model is allowed to generate, and how the system responds when retrieval returns weak evidence or inference exceeds budget.
The nonfunctional requirements are also broader now. Good candidates volunteer cost ceilings, token budgets, quality metrics, privacy constraints, and safety checks without waiting for a hint from the interviewer. They state how they would measure quality, such as answer relevance or hallucination rate, and they propose graceful degradation, such as returning cited search results when generation is disabled.
Note: Drawing more boxes is not the signal. Defending why those boxes exist, how they fail, and what fallback path activates is the signal.
The map below summarizes the categories interviewers now probe:
The final step is to compress the lesson into the mental checklist you should carry into the next chapter topic.
Conclusion
Classical System Design remains the base layer. You still need load balancers, queues, storage design, retries, partitioning, and fault isolation. AI changes the request path by inserting probabilistic inference, semantic retrieval, GPU-bound capacity limits, and quality-control layers on top of that base.
The broken assumptions are now clear. Compute is more expensive, latency is less predictable, outputs are less deterministic, and failure can mean wrong but confident rather than simply down. In interviews, strong candidates surface these trade-offs early and quantify them with latency, token, cost, and quality budgets. The next lesson builds on this technical shift and shows how different companies and roles evaluate these signals differently.