Search⌘ K
AI Features
Home
Courses
Grokking the Data Science Interview

Diagnosing an Underperforming Model

Explore how to identify key causes of model underperformance, such as data leakage, distribution shifts, label noise, and overfitting. This lesson guides you through validating evaluation setups, inspecting feature timing, splitting strategies, and auditing labels to ensure reliable model assessments in practical machine learning applications.

We'll cover the following...

A card-fraud classifier reports a ROC AUC of 0.99 on a random validation split, yet its ROC AUC falls to 0.61 for transactions from the newest week. The score looks strong until the holdout slice asks the question the model will actually face.

The evaluation summary below contains the contradiction we need to explain. The overall validation result invites a model change, but the newest-week result first requires a validity check.

random validation split     ROC AUC: 0.99
newest-week holdout         ROC AUC: 0.61
new-merchant slice          ROC AUC: 0.64

An interviewer might ask, “What would you check first?” A weak answer jumps to a more complex model or a different threshold. That answer treats the symptom as ordinary underperformance, even though an almost perfect random-split score makes the evaluation setup itself suspicious.

A stronger answer starts with the split, feature timing, and label process. We would list four suspects without choosing one yet:

  • Leakage occurs when training or evaluation uses information unavailable when the prediction must be made.

  • Distribution shift occurs when the inputs or their relationship to the label differ between training data and the evaluation slice.

  • Label noise occurs when recorded labels differ from the target we intend to predict.

  • Overfitting remains possible when the model learned patterns specific to its training sample.

The ordering matters. A model cannot repair a feature created after the transaction, a random split that hides a time change, or labels that have not matured. We can say, “Before changing the model, I would verify that features, labels, and splits reflect what was known at prediction time.”

If the interviewer pushes back that the random split was stratified, stratification only preserves class ...