Debugging an Agent Pipeline With Traces
Debug agent pipelines by preserving failure records and locating the earliest step that contradicts authoritative source state. Correlate instrumented spans with separately controlled application records to distinguish mapper defects from downstream model behavior. Reproduce failures, repair status handling in code, and reject unsupported claims at the acceptance boundary. Test capability, safety, and collector coverage separately, and reserve separate representative holdout cases before tuning. Require explicitly accepted operating owners, monitoring thresholds, operator review, manual fallback, and a disablement plan before returning the affected path to service.
Two Northstar operator drafts say a technician visit is complete. The service events say it is scheduled. The team found the discrepancy in seven recorded runs, which is enough to act on but not a production failure rate.
When a live agent misbehaves, the final answer is the least useful thing to look at: it tells you something went wrong, rarely where. A trace records instrumented steps within the run, and debugging comes down to one discipline. Find the first step whose output contradicts the source of truth, fix that step, and prove the fix with the evaluation set you already have.
Act before you debug. Withhold the affected read-only draft path, send its requests to operators, and preserve the records before changing anything.
How do we read a trace?
The Agents SDK traces every run by default. A trace is one end-to-end workflow. Its workflow_name names the logical workflow or app, such as “Northstar draft”; its trace_id uniquely identifies the run and is generated automatically if you don’t pass one; and an optional group_id links several traces from the same conversation.
Inside the trace, spans record individual operations with start and end times and a parent. The SDK wraps each Runner.run call in a trace and creates spans automatically: agent spans for each agent, generation spans for model calls, function spans for tool calls, plus guardrail and handoff spans. You view them as a timeline in the Traces dashboard, or send them to your own tracing backend.
Read a trace top-down. Start at the agent span, follow the generation spans to see what the model received and returned, and check each function span’s arguments and result. A handoff span shows where control passed to another agent.
The SDK only sees what the SDK runs. Your own code between steps, such as a mapper that normalizes tool output, appears only if you add a custom span or log it in an application record keyed by the same task ID. For a full picture, join the trace with the release and model configuration, the source version, the tool outcome, and any separately collected action evidence.
Northstar’s draft path has operator review and no ticket write or customer message. There is no approval step in this path, so don’t invent an approval span to make the timeline look complete.
Where is the first wrong step?
For task D-04, the preserved application record gives this sequence:
Authorized read: The source for
NS-452returns version7, event typevisit, statusscheduled.Application mapper: Mapper version
1tests only whether the event type isvisit. It drops the status and emitsvisit_completed=true.Handoff: The receiving specialist gets that false field. ...