The AI Product Engineering Lifecycle

Explore the AI product engineering lifecycle to understand how to build reliable AI features. Learn to define problems, design prompt contracts, evaluate results, diagnose failures, and improve features through monitoring and iteration within a practical loop.

We'll cover the following...

Should this even be AI?
The lifecycle, as a loop
Example: Summarizing support conversations
What’s next

In the previous lesson, we broke AI product engineering into seven ownership areas: the product problem, the model, context and data, application logic, outputs and actions, user experience, and quality and reliability. In practice, we don’t work through those areas once and move on. We cycle through them, and the results from one stage change what we do at the next. In this lesson, we’ll lay out that cycle as a single, repeatable workflow, then walk through it end to end with one real feature, from a first prompt to a fix driven by actual failures.

In this lesson, we will cover:

The one decision that comes before any of the others: whether a feature should use AI at all
The lifecycle as a loop, and where each stage is taught in this course
A full worked example: a support-ticket summarizer, built and fixed one stage at a time
A quick check to test whether you can diagnose a failure correctly

Should this even be AI?

Before any of the other ownership areas matter, we have to answer one question first: does this problem actually need a model at all?

Line 1: We import Python’s datetime module to parse and reformat the date string.
Lines 3–4: We define convert_date, which parses the input using the MM/DD/YYYY format and returns it re-formatted as YYYY-MM-DD.
Line 6: We call the function on a sample date and print the result to confirm it works.

The second case is a good fit for AI precisely because there’s no fixed mapping from input to output. That’s also what makes it harder to engineer: we can’t check the output against one right answer, which is why the rest of this lesson exists.

The lifecycle, as a loop

Once a feature is worth building with AI, we follow a repeating cycle:

Stage	What Happens	Where We’ll Learn It
Define the problem	Decide whether and where AI belongs, and what "good enough" looks like	Scoping an AI feature
Design the contract	Give the model the right instructions and context, and define a structured output our code can validate	Designing the model-application interface
Evaluate	Build a set of real test cases and measure how the baseline actually performs	Evaluating AI features
Add capabilities	Use what the evaluation revealed to decide whether the feature needs retrieval, tool calling, or model adaptation, and add only what's needed	Choosing additional capabilities
Harden and operate	Protect the feature from misuse, control what actions it can take, and monitor quality, latency, and cost	Operating in production

Notice how the loop closes at the last row: once we’re monitoring in production, new failure cases we find get added back to the evaluation set, and the cycle runs again. We treat evaluation as something that runs through every other stage, rather than something we finish after any single one of them.

Example: Summarizing support conversations

In the previous lesson, we used this feature as a one-line example: an AI assistant that turns a support conversation into a structured summary and a suggested next action. Let’s build it for real, one lifecycle stage at a time, and see exactly where it breaks and why.

Step 1: Define the problem

Looking back at the table above, this is a good fit for AI: the input is unstructured, and several summaries could reasonably count as correct. Before writing any code, we define what “good enough” means: every summary must name a category, suggest one next action, and flag itself for human review whenever the model isn’t confident.

Step 2: Design the contract

Instead of trusting whatever text the model returns, the application defines exactly what a valid response looks like, and forces the model to return exactly that shape.

The examples in this course run on Groq, which hosts open models like Llama 3.3 behind a free API key and an OpenAI-compatible interface, so every snippet here is something you can run yourself without a paid plan.

Python

import json
from groq import Groq
from pydantic import BaseModel
from typing import Literal
client = Groq(api_key="{{GROQ_API_KEY}}")
class TicketSummary(BaseModel):
    summary: str
    category: Literal["billing", "technical", "account", "other"]
    suggested_action: str
    needs_human_review: bool
SYSTEM_PROMPT = """You summarize a support conversation into a structured
record. Categorize it as billing, technical, account, or other. Suggest one
next action. Set needs_human_review to true if the conversation involves a
refund, a legal threat, or anything you're not confident about."""
RECORD_SUMMARY_TOOL = {
    "type": "function",
    "function": {
        "name": "record_summary",
        "description": "Record a structured summary of a support conversation.",
        "parameters": TicketSummary.model_json_schema(),
    },
}
def summarize_ticket(conversation: str) -> TicketSummary:
    response = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": conversation},
        ],
        tools=[RECORD_SUMMARY_TOOL],
        tool_choice={"type": "function", "function": {"name": "record_summary"}},
    )
    arguments = response.choices[0].message.tool_calls[0].function.arguments
    return TicketSummary(**json.loads(arguments))
if __name__ == "__main__":
    example = """Customer: I was charged twice for my subscription this month.
Agent: I'm sorry to hear that, let me look into it.
Customer: This is the second time this has happened. I want a refund."""
    print(summarize_ticket(example))

Lines 1–4: We import json for parsing the model’s response, the Groq client, and Pydantic’s BaseModel and Literal for defining the contract.
Line 6: We initiate the Groq client with our API key.
Lines 8–12: We define TicketSummary, the contract itself: every response must include a summary, one of four categories, a suggested action, and whether it needs human review.
Lines 14–17: We write the system prompt that tells the model exactly what to do and when to flag something for review.
Lines 19–26: We describe record_summary as a callable tool, using the TicketSummary schema directly as its parameters so the tool’s shape and the contract can never drift apart.
Lines 28–39: We define summarize_ticket, which sends the conversation to the model, forces it to call record_summary, and parses the returned arguments into a validated TicketSummary.
Lines 41–45: We run the function on a sample conversation and print the result, so the whole file executes and shows real output.

Step 3: Evaluate

A contract that's never been checked against real cases is still a guess. Say we'd collected 50 real past conversations, each with a category a human already agreed on, and run the baseline against all of them. Here's a small version of that same harness with three example cases, something we can actually run and see for ourselves, even though a real evaluation set would need far more than three:

Python

def load_real_conversations():
    # A real evaluation set would have 50+ cases pulled from actual past
    # conversations. These three stand in for that set.
    return [
        {
            "conversation": "Customer: My app keeps crashing when I open the camera.\nAgent: Let's try reinstalling it.",
            "expected_category": "technical",
        },
        {
            "conversation": "Customer: I don't recognize a charge on my card from you this month.\nAgent: Let me check your billing history.",
            "expected_category": "billing",
        },
        {
            "conversation": "Customer: I can't log into my account, it says my password is wrong.\nAgent: Let's reset it together.",
            "expected_category": "account",
        },
    ]
test_cases = load_real_conversations()
def run_evaluation(cases):
    passed, failures = 0, []
    for case in cases:
        result = summarize_ticket(case["conversation"])
        if result.category == case["expected_category"]:
            passed += 1
        else:
            failures.append(case)
    return passed, failures
passed, failures = run_evaluation(test_cases)
print(f"{passed}/{len(test_cases)} passed")

Lines 1–17: We define load_real_conversations, standing in for a real 50-case evaluation set with three example conversations, each paired with the category a human already agreed on.
Line 19: We load the test cases.
Lines 21–29: We define run_evaluation, which runs every case through summarize_ticket, counts how many match the expected category, and keeps the failures instead of discarding them.
Lines 31–32: We run the evaluation and print how many cases passed.

Suppose that’s roughly what a run against the full 50 turns up: the baseline passes most cases, but a cluster of failures shares one thing in common: every one of them mentions a refund policy specific to the customer’s subscription plan, information the prompt above never had access to.

Step 4: Add capabilities, only where the evaluation points

That's a gap the evaluation set actually measured, so we target the fix at exactly that gap: give the model the customer's plan policy before it answers.

Python 3.10.4

POLICY_DOCS = {
    "basic": "Basic plan subscriptions are refundable within 7 days of purchase.",
    "pro": "Pro plan subscriptions are refundable within 30 days of purchase.",
}
def get_refund_policy(plan: str) -> str:
    return POLICY_DOCS[plan]  # a real version would look this up from the team's own docs
def summarize_ticket(conversation: str, plan: str) -> TicketSummary:
    policy = get_refund_policy(plan)
    context = f"Relevant refund policy for this plan:\n{policy}\n\n{conversation}"
    response = client.chat.completions.create(
        model="openai/gpt-oss-120b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": context},
        ],
        tools=[RECORD_SUMMARY_TOOL],
        tool_choice={"type": "function", "function": {"name": "record_summary"}},
    )
    arguments = response.choices[0].message.tool_calls[0].function.arguments
    return TicketSummary(**json.loads(arguments))
if __name__ == "__main__":
    example = """Customer: I was charged twice for my subscription this month.
Agent: I'm sorry to hear that, let me look into it.
Customer: This is the second time this has happened. I want a refund."""
    print(summarize_ticket(example, plan="pro"))

Lines 1–4: We add POLICY_DOCS, standing in for our real policy documentation, keyed by plan.
Lines 6–7: We define get_refund_policy to look up the right policy for a given plan.
Lines 9–11: We update summarize_ticket to accept a plan, look up its policy, and prepend it to the conversation before anything else happens.
Lines 12–22: The model call itself is unchanged from Step 2; it now receives context (conversation plus policy) instead of the raw conversation.
Lines 24–28: We run the updated function on the same example conversation, now passing a plan, so the model has the information it was missing before.

When we re-run run_evaluation with this change, the refund-related failures clear up, since the model now has the information it was missing. If the failures had instead involved the model needing to open a refund directly in the billing system, an action rather than a fact, giving it a tool to call would have been the right fix. Either way, we let the evaluation set decide which capability to add.

Step 5: Harden and operate

Before shipping, we add three things that the evaluation set doesn’t cover on its own:

Validation: If the model’s response doesn’t match the TicketSummary schema, Pydantic raises an error instead of letting bad data through.
Human review: Whenever needs_human_review is True, we queue the summary for a person instead of applying it automatically.

Python 3.10.4

def queue_for_human_review(result: TicketSummary) -> None:
    print("Needs human review:", result)  # a real version would write to a review queue
def apply_summary(result: TicketSummary) -> None:
    print("Applying summary:", result)  # a real version would save it and notify the team
conversation = "Customer: I want a refund, this is unacceptable.\nAgent: I understand, let me help."
plan = "pro"
result = summarize_ticket(conversation, plan)
if result.needs_human_review:
    queue_for_human_review(result)
else:
    apply_summary(result)

Lines 1–2: We define queue_for_human_review as a stand-in for a real review queue.
Lines 4–5: We define apply_summary as a stand-in for whatever the application does with a trusted summary.
Lines 7–8: We set up one example conversation and plan to run through the whole pipeline.
Lines 10–14: We call summarize_ticket and route the result based on needs_human_review, the same contract field defined all the way back in Step 2.
Monitoring: We track how often a human corrects a summary after the fact. A rising correction rate is a signal, and we add every corrected case back into the evaluation set, which is how the loop closes.

Quiz

A team ships the baseline prompt for the support-summarizer feature, and it passes 40 of 50 cases in the evaluation set. All 10 failures happen when the customer asks about a refund policy specific to their subscription plan, information the model was never given. What should the team do next?

Switch to a larger, more capable model and keep everything else the same

Fine-tune the model directly on transcripts of refund-related conversations

Add retrieval so the model receives the customer’s specific refund policy before answering

Lower the evaluation bar so these refund cases count as passing

1 / 1

Dimension	Convert a date from MM/DD/YYYY to YYYY-MM-DD	Turn a messy support conversation into a summary and next steps
Input	A fixed, predictable format	Free-form text that varies every time
Output	Exactly one correct answer	Several different summaries could all be "good enough"
Right tool	A few lines of deterministic code	A model, with validation and review built around it

1.Introduction to the Course

2.Scoping an AI Feature

3.Designing the Model-Application Interface

4.Evaluating AI Features

5.Choosing Additional Capabilities

6.Operating in Production

7.Wrap Up