Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Why Accuracy Fails on Imbalanced Data

Understand why accuracy can be misleading for imbalanced datasets, such as fraud detection, by examining confusion matrices and recall scores. Explore balanced accuracy and precision-recall curves to better evaluate classifiers on rare events. Learn to interpret metrics that reflect both positive and negative class performance and make informed decisions about model thresholds and alert reviews.

We'll cover the following...

A payment-fraud classifier can report 98% accuracy while never flagging a single fraudulent transaction. If fraud occurs in 4 of 200 transactions, an always negative classifier correctly approves the other 196 transactions and misses every fraud case. The result looks strong only because the negative class dominates the data.

The base rate is the share of evaluated rows that belong to a class. Here, the positive-class base rate is 4/200=0.024/200=0.02, or 2%. Accuracy counts every correct negative prediction, so 196 true negatives can overwhelm four false negatives. For rare-event detection, the first debugging question is not whether accuracy is high. It is whether the confusion matrix contains any true positives.

Inspect the all-negative baseline

The held-out labels below represent 200 transactions with four confirmed fraud cases. The prediction array contains only 0 ...

Python
import numpy as np
from sklearn.metrics import accuracy_score, confusion_matrix, recall_score
def main() -> None:
    y_true = np.array([0] * 196 + [1] * 4)
    y_pred = np.zeros_like(y_true)
    acc = accuracy_score(y_true, y_pred)
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    rec = recall_score(y_true, y_pred)
    print(f"Accuracy: {acc:.2f}")
    print(f"Confusion matrix counts: TN={tn}, FP={fp}, FN={fn}, TP={tp}")
    print(f"Recall: {rec:.1f}")
if __name__ == "__main__":
    main()
...