Discovery and Scoping
Explore how to perform discovery and scoping for a text-to-SQL analytics agent in a global retail context. Learn to gather real user needs, request evidence, define measurable acceptance criteria, and negotiate a clear scope that aligns with business authority and operational constraints.
“Can a better model guarantee 85%?” asks the sponsor. “Before accepting that commitment, I need the actual questions, the pilot’s errors, the approved meanings, the access rules, and the consequences of a wrong answer,” you reply. Discovery turns those requests into a sales-and-inventory scope and a measurable acceptance contract. That contract gives your design something defensible to achieve while keeping an unmeasured target separate from demonstrated performance.
Before promising 85%
With the mandate established in the previous lesson, discovery now tests what users need and what evidence can support a commitment.
Ask five questions, each attached to evidence someone can produce:
What do users actually ask? Request the BI ticket log, original wording, follow-up exchanges, and delivered reports. These reveal the work hidden behind short prompts.
Why did the pilot fail? Request its test inputs, available outputs, and error reviews. Separate an incorrect business interpretation from an incorrect query.
Who owns key definitions? Request competing definitions, their business authorities, and existing approved reports. Agreement needs a decision path when teams disagree.
How are permissions enforced today? Request current user-to-region mappings, access arrangements, and examples of permitted and denied requests.
What happens after a wrong number? Ask which decisions depend on reports, what errors have cost, and who would receive and correct an incorrect published answer.
A useful discovery response names the evidence needed to decide. “We need more requirements” leaves the customer without a concrete next action.
In the interview, the interviewer usually plays the customer and answers your discovery questions as you ask them. Spend roughly the first quarter of the session on discovery, then propose a scope out loud, for example: "Based on what you've told me, I'd start with sales and inventory on curated tables, measure against real tickets, and decline when a definition is unapproved. Does that match your priorities?" Proposing a scope and inviting correction shows more judgment than asking questions indefinitely.
What discovery establishes
Discovery supplies findings that were open hypotheses in the previous lesson:
Real work: A log of 600 past BI tickets is available.
Schema: Names such as
fct_sls_dly_v3obscure meaning.Metric conflict: Three teams use different definitions of net sales; conflicting definitions caused most pilot errors.
Testing mismatch: The pilot was tested on synthetic questions rather than real ones.
Error consequences: Users prefer an explicit inability to answer over a confident wrong number because wrong answers are costly.
Preserve the separately reported historical outcome: About 38% correctness on real questions despite attractive demos. The discovery finding about synthetic testing does not make 38% a synthetic-test score. The record supplies no method or sequence for the later ...