Knowing When Not to Trust the Answer
Explore how to interpret AI model confidence as a critical signal for validating structured outputs. Learn to design prompts that request confidence levels and implement routing that accounts for both answer correctness and certainty. This lesson helps you improve reliability by distinguishing well-formed answers from trustworthy ones.
Every safeguard built so far in this chapter catches a specific, nameable kind of failure: malformed JSON, a value outside our schema, a field of the wrong type. None of those catch the perfectly well-formed response that passes every validation check and is simply a low-confidence guess dressed up in the exact shape we asked for. This lesson is about that gap, cases where the format is fine but the model itself isn't sure, and where the right move is asking it to say so.
The gap validation can't close
Go back to the TicketExtraction schema from two lessons ago: category restricted to four values, urgency a plain string, mentions_refund a boolean. A ticket that reads "something feels off with my account lately, not sure what" could plausibly get classified as account_access, and that response would pass every check we've built: right type, right enum member, right structure. Nothing about the shape tells us the model was genuinely confident versus essentially guessing between two or three plausible options.
This is a different category of problem from anything this chapter has addressed so far. Malformed JSON is a syntax problem. A category outside our enum is a values problem. A ...