ROC Curves, AUC, and Choosing a Classification Threshold

A churn classifier can assign each customer a score such as 0.82, then a retention policy turns that score into a decision. Offering a discount to a customer who would have stayed wastes discount budget. Failing to offer a discount to a customer who would churn can lose recurring revenue.

A classification threshold is the score cutoff that converts a model score into an action. With a threshold of 0.50, a customer scored at 0.82 receives the offer and a customer scored at 0.31 does not. Raising or lowering that cutoff changes the false positive and false negative counts.

Before choosing a cutoff, we may need to compare two models as rankers. A threshold-free comparison asks whether customers who churn tend to receive higher scores than customers who stay. A threshold-specific evaluation asks where to place the offer cutoff after accounting for the cost of each error.

Reading a ROC curve

Because each threshold produces a different confusion matrix, a receiver operating characteristic curve (ROC curve) plots the model’s behavior across thresholds rather than at only 0.50.

The vertical axis is the true positive rate (TPR), which is the same quantity as recall. It measures the share of actual churners that the policy targets.

The horizontal axis is the false positive rate (FPR). It measures the share of customers who would stay but still receive a retention offer.

At a very high threshold, few customers receive an offer. Both TPR and FPR are usually low because the policy catches few churners and wastes few offers. Lowering the threshold targets more customers. TPR can rise because the policy reaches more true churners, while FPR can also rise because it reaches more customers who would stay anyway.

A point on the ROC curve therefore represents one threshold and one pair of values, ...