Live Debugging an Almost-Right Pipeline
Explore how to methodically debug nearly correct AI pipelines during customer deployments and interviews. Learn to reproduce failures, trace tool inputs and outputs, identify the first point of error, apply fixes at the root cause, and safeguard with evaluation cases. This lesson provides a practical example debugging an expense audit agent and highlights how to narrate your reasoning effectively in technical interviews.
Forward-deployed work often starts with a customer system that mostly works. The demo passed, most outputs look fine, and a few decisions are quietly wrong. Technical interviews test the same skill, and an interviewer watching a candidate debug learns more from the method than from the speed of the fix.
A candidate who guesses, edits the prompt, and reruns once can get lucky. A candidate who reproduces the failure, traces it, and fixes it at its source shows the judgment a customer deployment needs. The examples follow one bug through an expense audit agent, from the first complaint to the eval case that keeps it fixed.
Why almost-right bugs are hard
An almost-right bug is a defect that leaves most outputs correct and makes the wrong ones look reasonable. Claude pipelines produce this kind of bug often for three reasons.
Outputs vary between runs: The same input can produce different results, so one good run proves little, and one bad run can look like noise.
Errors look plausible: Claude writes a fluent, confident reason around whatever input it receives. A wrong input produces a wrong decision with a convincing explanation.
The cause sits upstream: The symptom appears at the final decision, while the cause often lives in a tool, a data format, or a prompt several steps earlier.
The third reason matters most. Looking at the final answer and editing the prompt treats the symptom, so a reliable method has to walk back to where the problem started.
A five-step method for Claude pipelines
The method below works on any pipeline where Claude calls tools. Each step answers one question before the next step begins.
Reproduce: Run the failing input several times and count the wrong results, which separates a consistent bug from run-to-run variation.
Trace: Record every tool input and output exactly as they passed between Claude and the tools.
Find the first divergence: Walk the trace from the start and stop at the first value that differs from the source data.
Fix at the source: ...