Search⌘ K
AI Features
Home
Courses
Google FDE Interview Prep

Building an Evaluation Harness

Explore how to build an evaluation harness that measures agent performance against customer-approved criteria. Understand the importance of case contracts, controlled executions, and evidence retention to produce reproducible, defensible release decisions. This lesson helps you design metrics, validate qualitative judgments, and apply gates ensuring safe, compliant AI deployments.

We'll cover the following...

A policy-answering assistant improves from 90 correct outcomes in 100 cases to 94, but one execution now reads a restricted document before refusing the request. Would you release it? An evaluation harness must turn that evidence into a reproducible decision, preserving the distinction between customer-approved criteria, observed execution, and a fallible automated judge. That distinction determines whether a higher score actually supports shipping your own agent.

Release or block?

The release owner has already agreed that the assistant must never read material outside the caller’s permitted scope. Commit to a decision before inspecting the slice results.

Release or block?

Q1. The baseline has 90 correct outcomes in 100 cases; the candidate has 94/100. In one observed execution, the candidate reads a restricted document outside the caller’s permitted scope, then refuses the request.

The release owner has already agreed that the assistant must never read outside that scope. Before seeing the slice report, choose a decision and its decisive reason.

Do you find this helpful?

More runs can help diagnose the defect, but cannot make this candidate pass the agreed contract.

An evaluation harness combines versioned cases, controlled execution, retained observations, scoring, and release rules. A harness measures inspectable execution properties across requests and changes, producing a reproducible release decision. Its useful output is a defensible release report with evidence behind every decision.

Specify the case contract

A golden set is a versioned collection of cases with approved expected outcomes or assessment criteria, not necessarily one exact reference sentence. Faithful paraphrases, clarification, refusal, and handoff can all be correct.

For a travel-policy assistant, each case records:

  • Identity: Case ID, slice, and underlying task or source family.

  • Input: Request or conversation and runner-supplied trusted caller scope, never authority inferred from user text.

  • Environment: Source snapshot and injected tool responses, failures, or latency.

  • Outcome contract: Allowed terminal outcome, required evidence, mandatory operations, and forbidden operations.

  • Assessment: Customer-approved critical assertions and qualitative criteria.

For the worked fixtures, every case uses source snapshot policies-s17, caller employee-7, and scope allowing travel-current revision 3 while denying employee-restricted revision 2. The approved travel document says that destination determines the applicable allowance and that reimbursement requires preapproval. These are exercise policy facts, not Google policies.

The starter expands these shared fields and five records into complete JSON objects:

  1. routine-001: routine; ANSWER the Northport question from current policy; forbid restricted reads.

  2. ambiguity-001: ambiguity; CLARIFY the missing destination; forbid dependent lookup before clarification. ...