How AI Reshaped System Design and Interviews
Understand how AI inserts a new probabilistic dependency, the model call, into otherwise deterministic System Design. Learn why its cost, latency, and unpredictability break long-standing assumptions about compute, caching, and failure. Reason through the cost–latency–quality trade-off that now anchors every AI System Design and interview answer.
Picture a familiar interview question, like designing search or a support chat. The candidate starts confidently and draws a client, an API gateway, a cache, stateless app servers, a queue, and a sharded database. The diagram looks clean, scalable, and very 2019.
Then the interviewer asks where semantic retrieval happens, how answers are grounded in company documents, what the inference latency budget is, and how the system detects a fluent but fabricated reply. Suddenly, the whiteboard has missing boxes.
Those gaps have a common cause. For over a decade, System Design meant one thing, which was moving and storing data at scale, predictably. You used load balancers, caches, sharded databases, and queues, and you reasoned about latency, throughput, and availability. That toolkit is still the foundation. But if you last drew an architecture a few years ago, your reference picture is missing a box that changes everything around it.
That box is a model call. Your system now depends on an LLM or another large model while serving each request. It looks like just another service, but it isn't. Unlike every other component on the whiteboard, it is expensive, slow, and probabilistic, meaning the same input can produce different outputs. Those three properties quietly break the assumptions classic design was built on.
AI didn't add complexity everywhere. It added one dependency that is expensive, slow, and probabilistic, and together those properties create a single trade-off between cost, latency, and quality. Improving one usually costs you another. Interviewers now expect you to balance this trade-off out loud. Master that triangle, and the rest follows.
We aim to stay at the System Design level and skip model internals. The focus is what changed for architects and interview candidates, and subsequently explain how company and role expectations shifted in the next lesson.