Calibration of Predicted Probabilities

Analyzing model calibration and accuracy

One interesting feature of the previous lesson's figureCaption: Plot of default rate and sample count for equal-interval bins is that the line plot of default rates increases by roughly the same amount from bin to bin. Contrast this to the decile plot in the previous lessonCaption: Default rate according to model prediction decile, where the default rate increases slowly at first and then more rapidly. Notice also that the default rate appears to be roughly the midpoint of the edges of predicted probability for each bin. This implies that the default rate is similar to the average model prediction in each bin. In other words, not only does our model appear to effectively rank borrowers from low to high risk of default, as quantified by the ROC AUC, but it also appears to accurately predict the probability of default.

Measuring how closely predicted probabilities match actual probabilities is the goal of calibrating probabilities. A standard measure for probability calibration follows from the concepts discussed above and is called expected calibration error (ECE), defined as

where the index $i$ ranges from 1 to the number of bins (N), $$ ...