Building LLM Eval Sets
Explore how to construct and use targeted evaluation sets for large language models, focusing on assessing relevance, faithfulness, and proper citation in grounded question answering. Understand defining evaluation units, setting grading criteria, and maintaining dataset hygiene to detect regressions and improve system reliability.
We'll cover the following...
A retriever tweak seems better after three manual tries, then a different query type produces a confident answer with missing or wrong citations. The fix is not more vibes testing. The fix is a small evaluation set where each example makes a regression obvious because it pins down what the system was allowed to use and what the output must prove.
Scope is turning fuzzy quality for grounded Q&A with citations into a dataset we can run repeatedly.
Define the evaluation unit
With a minimal item in hand, the next step is choosing the evaluation unit so scores map to a real system behavior. For grounded Q&A, the input is usually the user question plus the retrieved context we want the model to rely on, captured as text chunks with stable source identifiers. The output is the answer plus whatever citation structure our app emits, such as inline markers or a citations array keyed by source_id. ...