Search⌘ K
AI Features
Home
Courses
Grokking the Data Science Interview

Practice Set: Metric Diagnosis Cases

Explore how to diagnose drops in key product metrics by defining appropriate metrics, performing integrity and funnel checks, and ranking hypotheses to explain changes. This lesson guides you through structured analysis of six real-world cases, helping you validate data, isolate causes, and recommend actionable insights for product and metric health.

We'll cover the following...

“Daily active users fell yesterday. What caused it?” A defensible answer does not invent a cause from one chart. We define daily_active_users, check whether the data can support the comparison, localize the movement, rank explanations, and name the next analysis that could disprove each explanation.

This set has six metric-drop cases of increasing difficulty. Cases 1–5 each have two hints and a structured answer; Case 6 adds a weak answer, a stronger answer, and two interviewer follow-ups. A true/false self-check closes the set. For each case, state the first check, two ranked hypotheses, and a result that would change the ranking before opening the hints. Use the chapter routine: clarify → define the metric → integrity checks → slices or funnel → ranked hypotheses → confirmatory analyses → recommendation. All figures are illustrative assumptions.

Problem 1: The disappearing active user event

Scenario: daily_active_users fell from an assumed 120,000 to 84,000 on Tuesday. The metric counts eligible signed-in accounts with at least one app_open event per calendar day. The decline appears only in the iOS client.

Question: What is the first check we should run, and which explanations should we rank to diagnose the iOS drop?

The drop is a step change on one platform. What kinds of change happen at a platform boundary overnight?

Solution

  • First, verify whether app_open still arrives from iOS. Compare raw event volume, distinct signed-in accounts, and iOS client versions before and after Tuesday. A sudden step change at one platform boundary is more consistent with instrumentation or release behavior than a gradual change in customer demand.

  • Rank a renamed or failed app_open event first. Compare app_open counts with an independent signed-in event such as screen_view or authenticated API requests. If iOS screen_view remains stable while app_open collapses, the metric definition no longer observes the same behavior.

  • Rank a real iOS access failure second. Slice authentication failures, crash rate, and session starts by client version. If those rates rise for the affected version, the decline may reflect a product failure rather than logging.

  • Recommend treating the top-line drop as untrusted until the event pipeline is reconciled. If independent activity signals also fall, escalate the product-access hypothesis. If they do not, backfill or repair the metric rather than claiming that engagement fell.

Problem 2: The denominator changed

Scenario: Purchase conversion fell from an assumed 4.0% to 3.2% week over week. The metric is purchasers / eligible_sessions. A release changed the definition of an eligible session on the same Monday. ...