Knowing When Not to Trust the Answer
Explore how to distinguish trustworthy AI model responses from low-confidence guesses despite valid formatting. Understand methods to have the model self-report confidence, enabling better routing decisions that separate certain outputs from ambiguous ones for human review.
Every safeguard built so far in this chapter catches a specific, nameable kind of failure: malformed JSON, a value outside our schema, a field of the wrong type. None of those catch the perfectly well-formed response that passes every validation check and is simply a low-confidence guess dressed up in the exact shape we asked for. This lesson is about that gap, cases where the format is fine but the model itself isn't sure, and where the right move is asking it to say so.
The gap validation can't close
Go back to the TicketExtraction schema from two lessons ago: category restricted to four values, urgency a plain string, mentions_refund a boolean. A ticket that reads "something feels off with my account lately, not sure what" could plausibly get classified as account_access, and that response would pass every check we've built: right type, right enum member, right structure. Nothing about the shape tells us the model was genuinely confident versus essentially guessing between two or three plausible options.
This is a different category of problem from anything this chapter has addressed so far. Malformed JSON is a syntax problem. A category outside our enum is a values problem. A confident-looking ...