Diagnosing a Metric Drop
Explore how to diagnose a drop in product conversion metrics by verifying metric definitions, validating data integrity, and localizing issues within user funnels. Learn to ask critical diagnostic questions, differentiate between measurement errors and actual behavioral changes, and prioritize plausible explanations for metric declines. This lesson equips you to conduct structured analysis and communicate findings clearly in data science interviews.
We'll cover the following...
Checkout conversion fell by an illustrative 8% week over week, from payment_success event, a changed eligibility filter, or a denominator that now includes ineligible sessions can create the same top-line decline without any customer behavior changing.
The interviewer asks, “Checkout conversion dropped 8% week over week. What happened?” We first lock down the metric contract from "Choosing Metrics and Guardrails." Which conversion fell, such as payment_success / start_checkout or payment_success / product_view? Which population counts in the denominator? Which dates define the two weeks, and did the release or reporting window change?
A weak answer says, “The new checkout flow probably caused friction, so I would inspect the redesign.” It fails because it turns timing into causation and skips the checks that establish whether the metric still measures the same population and event.
A stronger answer starts with three questions:
Did the event definition, logging pipeline, or dashboard ...