Automated LLM Evaluation
Explore automated methods to evaluate large language models by measuring relevance and faithfulness of responses. Understand the use of deterministic checks, heuristic similarity, and LLM-as-judge rubrics. Learn how to calibrate automated scores with human spot checks to ensure reliable evaluation outcomes.
We'll cover the following...
A user asks a grounded Q&A feature a question, and we get two answers that both sound plausible. One answer directly addresses the question, but it includes a confident statement that never appears in the provided citations. Another answer stays inside the cited text, but it only answers half the question and drifts into a side detail. Those two outputs fail in different ways, so a single good or bad label collapses useful information. Relevance measures whether the answer addresses the user’s question, while faithfulness measures whether the answer’s claims are supported by the provided sources and citations. The operational rule is simple. A single overall score is only meaningful if we also know which dimension dominates it. ... ...