Finding the First Divergence in a Run
Explore how to analyze AI coding agent runs by tracing events to find the first divergence from expected outcomes. Understand how trace contracts link agent actions, runtime behaviors, and verdicts to pinpoint failure causes. Learn to create failure packets that guide repairs and improve reliability across sessions.
We'll cover the following...
- What does an observable agent run reveal?
- Which layers belong in the same trace?
- Why isn’t the conversation transcript enough?
- What should one trace event contain?
- How do you locate the first useful failure?
- How should a failed run guide the next action?
- Does every tool call need a detailed span?
- Can you diagnose Wrenfold from the trace?
- What can you add to your own harness today?
- How do you test the Trace Contract?
- What’s next?
Session 2 resumed CLIN-547 from Wrenfold’s new checkpoint. It confirmed the revision, reproduced the waitlist failure, and put cancellation and promotion inside one database transaction. The next run ended with one line:
FAIL waitlist-promotion: expected promoted + 1 appointment; received pending + 0
The agent added a database retry. The same test failed. It rewrote the queue-ranking query next, even though the trace from the previous session had already proved that the correct patient was selected.
The terminal exposed the final symptom. It hid the path that produced it. Inside the transaction, cancellation removed one appointment, waitlist selection found patient wl-204, and appointment insertion created the replacement. The next update changed zero waitlist rows because it filtered by the new appointment ID instead of the selected waitlist ID. The transaction then rolled back, producing pending + 0 at the test boundary.
The model could repair that predicate. The harness gave it too little runtime evidence to choose the predicate as the next boundary. This is another reliability failure.
What does an observable agent run reveal?
An observable run connects intent, action, runtime behavior, and the Verdict that closes it. You can follow one feature from the scope contract through edits and commands into application events, then see which evidence caused the completion gate to pass or fail.
Ordinary terminal output answers isolated questions. A diff shows what changed, a test report shows which assertion failed, and application logs show selected events. None of them connects the whole run unless the harness adds shared identifiers and causal links.
A task trace is that connected record. It assigns one run_id to the session, one event_id to each recorded step, and a parent_id when one event ...