Search⌘ K
AI Features
Home
Courses
Grokking the Data Science Interview

Practice Set: Applied ML Reasoning

Explore applied machine learning reasoning by framing prediction tasks, choosing baselines and evaluation metrics, running sanity checks, and proposing experiments. Develop skills to assess models effectively, handle operational constraints like review capacity, and interpret results critically to support defensible decisions in real-world settings.

We'll cover the following...

Mixed prompts test whether we can make a defensible decision before choosing a sophisticated model. Use the routine from "How ML Rounds Are Scored" to frame the task, choose a baseline, select a metric, run sanity checks, then name the next experiment.

For every proposed change, state the predicted effect and the result that would weaken the hypothesis. “Retrain on recent data” is incomplete. “Retrain on recent data should improve the newest-week slice; if it does not, inspect label maturity and feature stability” can be tested.

For each scenario, answer aloud in the same order. Name the prediction target and analysis unit, choose a baseline, name the metric or decision rule, identify the first check, and propose one next experiment.

Each problem has two layered hints. Use a first hint only when the task framing remains unclear. Use the second hint only after committing to the prediction target or non-model decision. Neither hint substitutes for a defended answer. These hints preserve the order of a strong response. They narrow the next question to ask without selecting the model, metric, or diagnosis for us. Hint 2 points at the metric or check, so try without it first.

Problem 1: Review queue for disputed charges

A payments team can manually review at most 60 disputed card transactions per day. The current classifier uses a threshold of 0.70.

The following code prints the holdout confusion matrix for one day’s 1,000 transactions.

Python 3.14.0
tp, fn, fp, tn = 36, 84, 24, 856
print("Flagged for review:", tp + fp)
print("Precision:", round(tp / (tp + fp), 2))
print("Recall:", round(tp / (tp + fn), 2))
print("Accuracy:", (tp + tn) / 1000)
print("Accuracy if nothing is flagged:", (tn + fp) / 1000)

There are 120 true disputes. A false negative can lead to an unrecovered charge, while a false positive consumes reviewer time and may delay a legitimate payment.

Choose a baseline, an offline metric, and a threshold-selection rule. Explain why accuracy alone would not settle the decision.

Question: Which baseline, metric, and threshold rule should the team use, and why is accuracy not enough?

Follow-up question: After expansion to a new market, disputes become 1% of transactions rather than 12%. The review team still has capacity for 60 transactions per day. What changes in the metric readout and threshold decision?

The unit is one transaction, and the team can review only 60 a day.

Name the unit and the constraint first.

Solution

Review capacity makes this an operating-point problem

The prompt asks which individual transactions deserve review, so the unit is one transaction and the label is a disputed charge under the team’s defined outcome window. A simple baseline could rank transactions by a transparent risk rule, such as unusually large charges from a first-time merchant, or use logistic regression if the available features support it. The purpose of the baseline is to show whether a more flexible model earns its complexity.

The current threshold produces 36 true positives, 24 false positives, and 84 false negatives. The review queue holds 60 transactions, exactly matching the 36 plus 24 flagged cases. Precision is 36/60=0.6036/60=0.60 ...