Evals, Guardrails, and Debugging with Traces
Explore how to create programmatic evaluation sets, apply guardrails, and use execution traces to detect and debug issues in AI systems. Learn to separate capability and safety cases and defend system readiness by analyzing case results, guarding against prompt injections, and tracing failures to their root cause.
We'll cover the following...
- What goes into an evaluation set?
- Which checks are code, and which need a model grader?
- Why grade the trace and not just the answer?
- How does prompt injection reach an agent?
- Where do guardrails and moderation fit?
- How do we debug a failing run with traces?
- How does this sound in an interview?
- Try it yourself
- What’s next?
Northstar's new release candidate passed 6 of 8 evaluation cases, matching the current version's result. The candidate also produced better-quality drafts, so the team considered it ready for release.
A case-by-case comparison revealed a different result. The candidate passed a capability test that the current version had failed, but failed a safety test that the current version had passed. In case S2, an operator authorized to access only ticket NS-209 requested a summary. The candidate produced an acceptable draft. However, the execution trace showed that the application had successfully retrieved NS-777, a ticket belonging to another customer.
The overall score remained unchanged, but the candidate had traded an improvement in capability for a regression in access control.
An FDE should be able to explain how an AI system's behavior has been evaluated and what evidence supports its readiness for deployment. In this lesson, you'll build a programmatic evaluation set for Northstar, implement application-level guardrails and prompt-injection defenses, and use execution traces to investigate failing runs.
What goes into an evaluation set?
An eval set is a versioned collection of cases, checks that run in code, and a gate that blocks a release when a check fails. It lives in the repository with the application, so it reruns every time anything changes.
The cases come from the customer’s real work. The FDE sits with Northstar’s service operations team and picks sanitized examples from the categories they actually handle, including rare ones that hurt: routine notes, urgent notes, ambiguous notes, requests for tickets the operator can’t see, and evidence that belongs to the wrong ticket.
Cases split into two families, with separate thresholds:
Capability cases ask whether the system does the job well.
Safety cases ask whether it stayed inside its boundaries.
Keeping them apart stops many easy drafting passes from hiding one serious access failure, which is exactly what the 6-of-8 total did.
A holdout set is reserved for release decisions. It is split off before any tuning and access to it is restricted, so nobody adjusts prompts or grader wording against it.
Each case records four things.
Part | Contents |
Identity and input | Case ID, family, sanitized input, the operator, and the ticket the request is entitled to |
Expected behavior | The expected route or label, and a rubric reference where meaning needs judgment |
Permitted operations | Allowed tools, forbidden effects, call limits |
Observed behavior | Final output, the trace of tool calls, and a separately collected record of side effects |
The expected entitlement comes from the case’s own fixture, never from the draft or the agent that produced it. And the side-effect record only means something if the collector was running. Missing instrumentation makes a case incomplete, not clean.
Which checks are code, and which need a model grader?
Anything that can be decided from trusted records gets a code grader: which tools were called, whether each read was authorized for this request, whether evidence matches the right ticket, whether call limits held, whether any forbidden write or message happened. Code graders are cheap and give the same answer every run. ...