Search⌘ K
AI Features
Home
Courses
Grokking the Data Science Interview

Evaluation Metrics That Matter

Understand how to evaluate machine learning models for rare event detection, like fraud, by using key metrics such as precision, recall, and F1 score. Learn why accuracy alone can be misleading and how to choose thresholds and metrics based on business costs and review capacity.

We'll cover the following...

The interviewer asks, “Flag fraudulent transactions. Only a tiny fraction are fraud, and manual review is expensive. How would you evaluate the model?” A weak answer is “I would maximize accuracy.”

To see why that fails, suppose a fraud classifier flags 20 of 10,000 transactions when 100 transactions are actually fraudulent. Ten flagged transactions are fraud, so the confusion matrix contains TP = 10, FP = 10, FN = 90, and TN = 9890. The classifier gets 9,900 decisions correct, which makes its accuracy look strong.

Accuracy answers the wrong question when fraud occurs in only 1% of transactions. A classifier that flags nothing also labels all 9,900 legitimate transactions correctly and reaches 99.0% accuracy, while catching no fraud at all.

The weak answer lets the 9,900 legitimate transactions dominate the score, so missed fraud barely changes accuracy. A stronger answer begins with error costs. A false negative lets a fraudulent transaction pass. A false positive sends a legitimate transaction to manual review. If missed fraud is the larger harm, we need to measure how much fraud the model catches, while still tracking the review burden.

Precision is the share of flagged transactions that are truly fraudulent. Recall is the share of all fraudulent transactions that the model flags.

The model’s 50% precision means half of reviews find fraud. Its 10% recall means 90 of 100 fraudulent transactions pass through. The harmonic mean of precision and recall, F1 score, is about 0.17. F1 falls because both the caught-fraud rate and flagged-item quality matter.

A candidate can say, “Accuracy is 99%, but it matches the always-negative baseline. The model catches only 10% of fraud, so I would evaluate recall alongside precision or review capacity.”

Files
main.py
README.md
Python 3.14.0
# Evaluation Metrics That Matter (Worked Example)
Scenario: 10,000 transactions with only 100 fraud cases (1% prevalence).
A fraud classifier flags 20 transactions.
Fixed confusion-matrix counts used by the lesson:
- TP = 10 (flagged and truly fraud)
- FP = 10 (flagged but legitimate)
- FN = 90 (not flagged but fraud)
- TN = 9,890 (not flagged and legitimate)
Key computed metrics from these counts:
- Accuracy = (TP + TN) / total = (10 + 9,890) / 10,000 = 99.0%
- Precision = TP / (TP + FP) = 10 / 20 = 50.0%
- Recall = TP / (TP + FN) = 10 / 100 = 10.0%
- F1 = harmonic mean of precision and recall ≈ 0.17
Why accuracy is misleading here:
- With severe class imbalance, the 9,900 legitimate transactions dominate accuracy.
- A classifier that *never flags anything* still gets 9,900/10,000 correct, i.e., 99.0% accuracy,
  while catching 0% of fraud.
This is why fraud evaluation is typically discussed in terms of precision/recall tradeoffs,
recall at a fixed review capacity, or cost-sensitive thresholding, rather than raw accuracy.

In main.py file:

  • Lines 1–4: The ...