Evaluation and Observability
Explore how to measure agent quality across task success, groundedness, tool use, safety, latency, and cost. Learn to build golden sets, enforce regression gates before release, trace agent behavior in production, and turn every failure into a lasting test case.
A traditional software system can be tested with fixed inputs and expected outputs, and once a test passes, it keeps passing until someone changes the code. Agents do not behave this way. The same question can produce different answers, a small prompt change can improve one type of request while breaking another, and a model version update can shift behavior across the entire system. Most failures are also silent. An agent that retrieves the wrong policy and answers confidently does not crash or raise an error. It simply gives a wrong answer that looks right.
For these reasons, evaluation and observability are what turn an agent demonstration into a production system a customer can trust. FDE interviewers for Google’s GenAI track pay particular attention to this area, and a design that treats evaluation as a final step is one of the most frequently flagged gaps in system design rounds. This lesson covers golden sets, regression gates, tracing, and a simple loop for debugging agent failures.
What to measure
An agent’s quality has several dimensions, and a single accuracy score hides important differences between them. A useful evaluation plan measures each dimension that matters for the use case.
Dimension | Question it answers | Example measure |
Task success | Did the agent solve the user’s problem? | Percentage of cases with a correct final outcome |
Groundedness | Is the answer supported by the retrieved sources? | Share of claims backed by a cited passage |
Retrieval quality | Did retrieval return the right information? | Whether the correct document appears in the top results |
Tool use | Did the agent call the right tools with the right arguments? | Tool selection and argument accuracy |
Safety and policy | Did the agent stay within allowed behavior? | Rate of refusals on out-of-scope or harmful requests |
Latency and cost | Is the agent fast and affordable enough? | Response time percentiles and cost per request |
Measuring these separately makes failures easier to locate. If task success drops but retrieval quality stays high, the problem is likely in reasoning or tool use, not in the data pipeline.