Choosing Between Classical ML and LLMs
Explore how to evaluate and choose between classical machine learning classifiers, embedding-based methods, and prompted large language models for routing support tickets. Understand performance metrics like accuracy, latency, and operational cost while learning to defend design decisions using concrete evidence and tuning considerations. This lesson helps you apply practical methods to meet quality and latency constraints in system design scenarios.
Route incoming support tickets into six stable queues. Require at least 96.5% correct routing and end-to-end p95 latency no greater than 700 ms.
That requirement calls for model selection: choosing a mechanism using task evidence, available data, latency, and operating cost. The required output is a queue identifier; searching resolved tickets and drafting a reply are separate decisions.
In an interview, defending those boundaries and calculating the resulting pipeline’s performance matters more than proposing the newest model.
Define the output
Fix six queue identifiers, Q1 through Q6, and their ownership definitions before comparing candidates. Correctness means selecting the reviewed target queue for a ticket. A syntactically valid identifier can still be wrong.
Separate the surrounding work:
Required-field checks: Rules check whether specified fields exist and satisfy exact constraints.
Queue prediction: A supervised classifier learns a mapping from ticket features to one of six targets.
Related-ticket search: Embeddings represent similarity and help retrieve related examples.
Response drafting: A prompted LLM interprets instructions and generates text from the supplied information.
These capabilities have different purposes. Begin an interview answer with the output contract and acceptance constraints, rather than committing to “Gemini everywhere.” Choose classifiers, embeddings, prompted models, tuned models, and hybrids by accuracy, latency, cost, and available data volume.
Establish four baselines
For this router, investigate four initial candidates:
Rules: Route an explicit, validated queue code directly. Rules need specified conditions rather than training labels; measure how much traffic those conditions cover and how often they are correct. Expanding keyword rules across varied language creates brittle exceptions.
TF-IDF plus linear classification: TF-IDF represents words using their frequency within a document and across the training collection. A linear classifier learns weights connecting those features to reviewed queue labels. It directly matches this fixed-label task and offers a useful low-latency baseline.
Embedding nearest-neighbor routing: Embed the ticket, retrieve similar reviewed training examples, and derive a queue from their labels. Shared vocabulary can conceal different ownership: Two tickets about invoices might concern a payment failure and an account-access failure. ...