Fraud Detection: Evaluation & Adversarial Dynamics

With the three-component architecture from the previous lesson (gradient-boosted trees, GraphSAGE embeddings, and a rule engine) operating within a latency budget, the next interview question is inevitable: how do you know the system actually works?

Consider a payment processor handling one million transactions per day. At a typical fraud rate of 0.1%, only 1,000 of those transactions are fraudulent. A model that labels every single transaction as legitimate achieves 99.9% accuracy while catching exactly zero fraud. Accuracy is meaningless here.

But the evaluation challenge in fraud goes deeper than class imbalance. Unlike image classification or sentiment analysis, fraud detection operates in an adversarial environment where the data distribution shifts because fraudsters actively study and adapt to the model’s decisions. This makes evaluation a moving target.

Interviewers expect you to address three evaluation pillars. First, asymmetric cost-aware metrics that reflect the real financial impact of errors. Second, temporal evaluation methodology that avoids data leakage from the future. Third, adversarial robustness through continual learning that keeps the model current as attackers evolve. A false negative (missed fraud) at a processor like Stripe can cost $500 or more per transaction in chargebacks and investigation overhead. A false positive (a blocked legitimate transaction) costs roughly 2–5 dollars in lost revenue and customer friction. That asymmetry drives every design decision that follows.

Precision-recall under asymmetric costs

The cost matrix that changes everything

Precision and recall carry fundamentally different business consequences in fraud detection. A false negative means a fraudulent transaction was approved, triggering chargeback liability, manual investigation costs, and potential regulatory penalties. A false positive means a legitimate customer’s transaction was declined, causing minor revenue loss and friction. The cost difference between these two error types is typically 50x to 100x.

This asymmetry reshapes how you select an operating threshold. A model tuned for high precision minimizes false positives but misses more fraud. A model tuned for high recall catches nearly all fraud but declines too many legitimate customers. The optimal operating point depends not on maximizing F1, but on minimizing total cost.

The cost-sensitive evaluation formula captures this directly:

where $C_{fn}$ is the cost per missed fraud and $C_{fp}$ ...