Search⌘ K
AI Features
Home
Courses
AI for Backend Engineers

Testing Code that Isn't Deterministic

Explore how to effectively test AI backend functions that include nondeterministic model calls. Understand separating deterministic from nondeterministic parts, using mocks for controlled testing, applying fixed input fixtures for parsing validation, and leveraging loose assertions in golden cases to verify realistic model responses.

We'll cover the following...

Every function built across this course calls a model somewhere inside it, and every one of those calls can return something slightly different each time. That makes a test like assert result == "some exact expected string" the wrong tool for most of what we have built, since that exact string might never come back again, even from a correctly working system. This lesson is about testing this kind of code honestly, by testing the parts that are actually deterministic directly, and testing the parts that are not with assertions loose enough to match what "correct" actually means for them.

One note before the code: this lesson writes its tests as plain functions with manual pass and fail printing, rather than using a separate test runner. That keeps this file runnable the same way every other lesson has been, with python main.py, and nothing extra to install or configure.

Setup code (given, not the focus of this lesson)

The block below is carried over so this file runs entirely on its own: the schema a valid result must match, the prompt builder, the code fence stripper, and the pipeline tying it all together. One thing is different here on purpose. call_llm's mock now branches based on a keyword in the prompt, rather than always returning the exact same fixed response, since this lesson's golden case tests specifically need to see different, realistic-looking results for different kinds of feedback.

import json
from enum import Enum
from pydantic import BaseModel, Field, ValidationError
class Sentiment(str, Enum):
    POSITIVE = "positive"
    NEUTRAL = "neutral"
    NEGATIVE = "negative"
class FeedbackCategory(str, Enum):
    PRODUCT_QUALITY = "product_quality"
    CUSTOMER_SERVICE = "customer_service"
    PRICING = "pricing"
    OTHER = "other"
class Confidence(str, Enum):
    HIGH = "high"
    LOW = "low"
class Feedback(BaseModel):
    sentiment: Sentiment
    category: FeedbackCategory
    severity: int = Field(ge=1, le=5, default=1)
    confidence: Confidence
    summary: str
def build_feedback_prompt(feedback_text: str) -> str:
    return (
        "Analyze the customer feedback below. Respond with a JSON object "
        "in exactly this shape, and nothing else:\n\n"
        "{\n"
        '  "sentiment": <one of: "positive", "neutral", "negative">,\n'
        '  "category": <one of: "product_quality", "customer_service", '
        '"pricing", "other">,\n'
        '  "severity": <integer from 1 to 5>,\n'
        '  "confidence": <one of: "high", "low">,\n'
        '  "summary": <string, one short sentence>\n'
        "}\n\n"
        f'Feedback: "{feedback_text}"'
    )
def build_repair_prompt(original_prompt: str, bad_response: str, error: ValidationError) -> str:
    return (
        f"{original_prompt}\n\nYour previous response did not match the "
        f"required format. Previous response:\n{bad_response}\n\n"
        f"Validation error:\n{error}\n\nRespond again with only the "
        "corrected JSON object."
    )
def strip_code_fence(text: str) -> str:
    stripped = text.strip()
    if stripped.startswith("```"):
        lines = stripped.split("\n")[1:]
        if lines and lines[-1].strip() == "```":
            lines = lines[:-1]
        return "\n".join(lines).strip()
    return stripped
# USE_MOCK is True by default, so this file runs immediately with no
# API key. Set it to False and fill in a real provider block below to
# use an actual model. The mock branches on a keyword in the prompt so
# this lesson's golden case tests have something realistic to check.
USE_MOCK = True
def call_llm(prompt: str) -> str:
    if USE_MOCK:
        lowered = prompt.lower()
        if "charged" in lowered or "twice" in lowered:
            return (
                '{"sentiment": "negative", "category": "pricing", '
                '"severity": 4, "confidence": "high", '
                '"summary": "Customer was charged twice this month."}'
            )
        if "crashes" in lowered:
            return (
                '{"sentiment": "negative", "category": "product_quality", '
                '"severity": 4, "confidence": "high", '
                '"summary": "App crashes when opening saved documents."}'
            )
        return (
            '{"sentiment": "negative", "category": "customer_service", '
            '"severity": 4, "confidence": "high", '
            '"summary": "Support took four days to respond."}'
        )
    # from openai import OpenAI
    # client = OpenAI(api_key="your-openai-api-key")
    # response = client.chat.completions.create(
    #     model="your-chosen-openai-model",
    #     messages=[{"role": "user", "content": prompt}],
    # )
    # return response.choices[0].message.content
    raise NotImplementedError("Fill in a real provider block, or set USE_MOCK back to True.")
def get_validated_feedback(feedback_text: str, call_llm=call_llm, max_attempts: int = 2) -> Feedback:
    prompt = build_feedback_prompt(feedback_text)
    current_prompt = prompt
    for attempt in range(1, max_attempts + 1):
        raw_response = call_llm(current_prompt)
        cleaned = strip_code_fence(raw_response)
        try:
            parsed = json.loads(cleaned)
        except json.JSONDecodeError:
            continue
        try:
            return Feedback.model_validate(parsed)
        except ValidationError as error:
            current_prompt = build_repair_prompt(prompt, cleaned, error)
    return Feedback(
        sentiment=Sentiment.NEUTRAL,
        category=FeedbackCategory.OTHER,
        severity=3,
        confidence=Confidence.LOW,
        summary="Could not be automatically assessed.",
    )

One detail worth noticing here, since it matters for everything that follows: get_validated_feedback now takes call_llm as a parameter, with the module-level call_llm function as its default value. This is the same dependency injection pattern from the service module lesson earlier in this course, and it is exactly what makes swapping in a test double possible without touching this function's own code at all. ...