Grading and Scoring Model Output
Explore effective methods for grading AI model output, including exact match for fixed categories, rubric-based grading for free text, and automated judging with models. Understand common pitfalls and learn how to design clear, checkable criteria to maintain evaluation accuracy and reliability.
The previous lesson built an evaluation set with a known-correct answer for every case. It never addressed how “known-correct” actually gets checked against what the model returned. Every evaluation run so far has checked one thing, whether category exactly matches. This lesson covers what to do about everything else the support-summarizer produces, summary and suggested_action, where there’s no single correct string to match against.
In this lesson, we will cover:
Why exact match works for some fields and fails immediately on others
Grading free text with a rubric, and why a vague rubric fails the same way a vague prompt does
Automating rubric grading with another model call, and why that judge isn’t automatically trustworthy either
Which ...