Evaluating and Guardrailing LLM Output
Explore how to build a small frozen evaluation set that covers ordinary, ambiguous, unsupported, and adversarial cases. Understand how to judge retrieval, answers, tool handling, and cost separately, and why permission checks must hold even when the model misbehaves, preparing you to make defensible quality claims.
Synthetic evaluation results for an expense assistant:
Variant A passes 6 of 8 answer tests.
Variant B passes 7 of 8.
Variant B has a higher overall answer pass rate. Before recommending Variant B, compare the failed test cases, permission enforcement, and cost per response.
The fictional customer requires two capabilities: explaining expense policies and retrieving claim statuses for the authenticated user. The MVP does not support claim approvals or employee-data exports.
Evaluate retrieval quality, answer correctness, application-enforced permissions, and operational performance independently before calculating an overall score.
Building the evaluation set
An evaluation case specifies inputs and success criteria. A trial is one attempt at that case. A trace records what happened during the attempt; an action’s outcome is its actual effect in application state. An assistant saying “approved” does not establish that a claim became approved.
Freeze this policy and its identifiers:
meals_v1: Meals are reimbursable up to $40 per person.lodging_v1: Lodging requires an itemized receipt.The supplied policy says nothing about pet boarding.
Keep access_v1 as a separate access-policy reference stating that users may view only their own claim status. Enforce this restriction using authenticated application identity and verified claim ownership, not policy text alone.
The authenticated session identifies user U7. Claim C17 belongs to U7, while claim C88 belongs to U9. Reset the fixture to the same initial state before each test case. ...