Automated LLM Evaluation
Explore techniques for automated evaluation of large language models by distinguishing relevance from faithfulness. Understand how to implement scoring rubrics, apply both deterministic and heuristic checks, and calibrate metrics with human oversight to ensure reliable model assessment.
We'll cover the following...
A user asks a grounded Q&A feature a question, and we get two answers that both sound plausible. One answer directly addresses the question, but it includes a confident statement that never appears in the provided citations. Another answer stays inside the cited text, but it only answers half the question and drifts into a side detail.
Those two outputs fail in different ways, so a single good or bad label collapses useful information. Relevance measures whether the answer addresses the user’s question, while faithfulness measures whether the answer’s claims are supported by the provided sources and citations. The operational rule is simple. A single overall score is only meaningful if we also know which dimension ...