Building an Evaluation Set
Establish an evaluation set to determine whether an agent is ready for release, moving beyond subjective assessments. Include real-world cases and automated checks to measure performance against defined benchmarks and verify both capability and safety. Categorize cases into capability and safety families, and reserve a holdout set for final release decisions. Use code-based and model-based graders with strict criteria to assess outputs and trace events. Maintain evidence integrity, apply version control, and coordinate across teams to support reliable, evidence-based release decisions.
“How would you know this agent is ready to ship?” Every FDE design discussion reaches that question, and “it looked good in testing” isn’t an answer. The answer is an evaluation set: real cases, checks that run in code, and a gate that blocks a release when a check fails.
An evaluation measures performance against benchmarks for a defined use case, and expert feedback can help establish accuracy and relevance. For Northstar, the benchmark must cover what the system did as well as what it said, because a convincing draft can follow an unauthorized read. Keeping the evaluation set in your code makes that check repeatable every time the workflow changes.
Which cases belong in the set?
Start from the scope you already contained. The read-only triage workflow may read an authorized record and write a draft for operator review. It may not write to a ticket or message a customer. The callback example from the previous lesson lives in a separate sandbox branch, so testing it doesn’t add callbacks to the triage release.
Sit down with service operations and pick sanitized cases from the categories their team actually handles, including the rare ones that hurt:
Routine and urgent notes: Does the system choose the expected route or label?
Ambiguous notes: Does the draft explain the uncertainty without inventing a resolution?
Wrong-ticket evidence: Is mismatched evidence rejected before a specialist receives it?
Denied reads: Does the request go to manual review without exposing protected content?
Unknown callback outcomes: In the sandbox branch, does the system keep the operation ID and escalate when status is unavailable?
Split these into two families. Capability cases ask whether the system does the job well; safety cases ask whether it stayed inside its boundaries. Keep them separate so that many easy drafting passes can’t hide one serious access failure.
A holdout is a set of cases reserved for the release decision. Split it off before any tuning and restrict who can see it, so nobody adjusts the agent or the grader wording against its results. The eight Northstar cases used later in this lesson are a development set for debugging, not that holdout. First, match these four example cases to their family or partition.
The ambiguous-draft case measures task quality. The unauthorized-read case tests an access boundary. They answer different questions, so they get different thresholds.
What does each case record?
A useful case ties the input and the expected behavior to context the system under test can’t change. Version the case set, and give every case four parts:
Identity and input: ...