Deciding Whether an AI Feature Is Worth Building
Explore a structured approach to determine whether an AI feature should be built by applying four critical questions. Understand when AI adds value, how to avoid common mistakes, and make confident decisions that save time and resources before development.
In the previous lesson, we treated “should this even be AI?” as the first decision in the lifecycle, using one example pair to make the point. That’s not enough to actually make this call on a real feature. In this lesson, we build a framework with four concrete questions, ground each one in a real example, then apply the whole framework across several realistic candidates, so we walk away with a repeatable way to decide rather than a memory of one example.
In this lesson, we will cover:
Why this decision has to come before any of the other ownership areas
Four questions that decide whether a feature needs AI, each with its own example
How the same framework separates well-known AI products from features that shouldn’t use AI at all
The three ways engineers most often get this decision wrong
Why this decision comes first
When we work on the contract, the evaluation set, the architecture, or production hardening, from the first lesson’s ownership areas, we’re already assuming a model belongs in the feature. None of that work is worth doing if we got this decision wrong to begin with, which is why we get it right here, before anything else. Get it wrong in one direction and we’ve spent weeks building an evaluation set and a contract around a feature that never needed a model. Get it wrong in the other direction and we ship something that hallucinates its way through a problem a database lookup would have solved perfectly.
Four questions that decide this
Working through these four questions in order covers the ground.
Is there already a deterministic way to solve this? If a clear rule or a few lines of code already handle every case reliably, a model adds cost, latency, and a new way to be wrong, without buying anything in return. A feature that looks up a customer’s current account balance already has exactly one correct answer sitting in a database; there’s nothing to interpret, so there’s nothing for a model to add.
Does solving it require judgment on input that varies or is ambiguous? This is where a model’s ability to interpret unstructured input actually pays off. Fixed formats and exact-match problems don’t need that ability; free-form language, tone, and intent usually do. Deciding whether a product review is positive or negative isn’t a fixed-rule problem; the same words can cut either way depending on sarcasm and context, which is exactly the kind of judgment a rule struggles with and a model handles naturally.
Can the model be grounded in real information, or would it be guessing? A model answering from whatever it memorized during training is a fundamentally different, and riskier, thing than a model answering from information we actually hand it. A feature that answers “has my order shipped yet” can be grounded directly in the order database; a feature that answers “will this shipping carrier be reliable this holiday season” has no such source to hand the model, so it would be guessing no matter how the prompt is written. If the answer depends on facts we have and the model doesn’t, we can usually solve that by grounding the model in those facts, which is what the RAG chapter later in this course is for.
Is the benefit worth the new cost and failure modes? Even a genuine judgment call might not be worth automating yet. A simpler partial solution that only needs to catch most cases can be good enough, and “good enough for now” is a legitimate answer. Auto-tagging a photo with its dominant color could technically use a model, but a basic color-histogram calculation gets the same result in milliseconds, for a fraction of the cost, with no risk of a hallucinated answer.
The same framework, seen in real products
This shows up in real products too.
Perplexity passes all four questions; there’s no fixed rule for what someone actually wants to know from a search query, so it needs judgment; it grounds its answers in retrieved sources instead of guessing, and the value, a direct answer with citations instead of a page of links, justifies the cost.
GitHub Copilot passes as well; predicting the next few lines of code requires interpreting the surrounding file and cursor position, something no fixed template could do across every codebase a developer might have open.
Contrast that with a credit card number check on a checkout form. Checking whether a 16-digit number passes the Luhn checksum is a single, well-known deterministic algorithm that’s been correct for decades. A team that routed this through a model would be trading a fast, free, always-correct check for a slower, costlier one that’s occasionally wrong, exactly the failure mode the first question exists to catch.
Applying the framework to real features
Here’s what those four questions look like applied to five realistic features, side by side.
Candidate Feature | Verdict | Why |
Convert a date string to ISO format | Don't use AI | A deterministic function already handles every case correctly |
Summarize a support conversation into a next action | Use AI | Judgment on unstructured input, grounded in the conversation itself |
Extract a phone number from free-form text | Don't use AI, at least not yet | Regex already catches nearly every real case; the rare miss isn't worth a model's added cost and latency |
Answer "what's included in the Pro plan" from product docs | Use AI | Judgment on how the question gets phrased, grounded in the actual docs instead of a guess |
Decide whether a conversation needs escalation to a manager | Use AI, paired with human review | A real judgment call with no fixed rule set, but a wrong call is costly enough to keep a person in the loop |
Notice the middle three rows aren’t a flat yes or no. The phone-number case shows a deterministic solution that’s good enough today, worth revisiting only if it starts missing real cases. The escalation case shows AI being the right call and still needing a safety net, which is exactly what the next lesson builds out.
Three ways engineers get this wrong
Even with the framework in hand, the same three mistakes show up again and again.
Reaching for AI when a deterministic solution already works: Validating that an email address is correctly formatted is a solved problem; a single regex or a standard library call handles it. Routing that through a model instead adds latency, API cost, and a new way to be wrong (the model could call a valid address invalid), for no benefit over the one-line check it replaced.
Trying to hand-code a problem that actually needs judgment: It’s tempting to solve “should this conversation be escalated” with keyword rules, flagging words like “furious,” “lawsuit,” or “refund.” Tone, sarcasm, and context keep slipping past those rules, and every fix adds another condition that breaks something else. This is exactly the kind of call a model handles more robustly than a growing pile of hand-written conditions ever will.
Deciding once and never revisiting it: The phone-number example above is a real “don’t use AI, at least not yet.” Products change. New input formats show up, users start typing things the original regex never anticipated, and a check that was genuinely good enough can quietly stop being good enough. Revisiting this decision as the data changes is part of the ongoing job.
Quiz
A team wants a feature that decides whether a job candidate’s resume matches a job posting’s required skills, both listed as plain bullet points. Exact keyword matching misses valid matches, since “led a team” and “management experience” mean the same thing but share no words. Which of the four questions above makes this a good fit for AI?
Determinism, since there’s already a reliable rule for this today
Judgment, since matching requires interpreting different phrasings of the same skill
Grounding, since the model would be answering from what it memorized in training
Value versus cost, since automating this clearly isn’t worth the added latency
Apply it to a feature of your own: Pick a feature idea, one we’ve actually seen proposed at work or noticed in a product we use, and run it through the four questions above. Write down the verdict and the one question that decided it. “Don’t use AI, at least not yet” is just as valid an outcome as “yes, build it.” A defensible answer either way is what this framework is actually for.
What’s next
Deciding a feature is worth building with AI isn’t the end of scoping. The next lesson covers what happens when the model gets it wrong anyway, when to have it say so, when to ask the user, and when to bring in a person.