Search⌘ K
AI Features
Home
Courses
AI for Backend Engineers

Latency and Cost as Design Constraints

Explore how to use latency and cost as key design factors when calling AI models from Python backends. Understand how to estimate token usage and costs upfront, choose the right model size for your task, and decide when an AI call is necessary to build efficient, reliable backend AI features.

We'll cover the following...

Previous lessons in this chapter assumed the AI call was justified and focused on how to implement it correctly. This lesson asks an earlier design question: should we make the call at all, and if so, which model should we use? Latency and cost should not be treated as concerns that are addressed only after the feature works. They are design constraints that should influence model choice, request frequency, and overall architecture from the start, just as we account for database query cost and network round-trips in backend design.

Estimating tokens before we send anything

The previous chapter's lesson on reading responses showed us how to get exact token counts after a call completes. Before a call happens, we often want a rough estimate, enough to decide whether a request is likely to be expensive or slow, without needing to make the call just to find out.

Python
def estimate_tokens(text: str) -> int:
    """
    A rough estimate, not an exact count. Real tokenization varies by
    provider and by the specific text, but roughly 4 characters per
    token is a reasonable planning approximation for English text.
    """
    return max(1, len(text) // 4)
system_prompt = (
    "You are a support-ticket classifier. Categories: billing, technical, "
    "account_access. Respond with only the category name."
)
ticket_text = (
    "I've tried resetting my password three times now and the reset link "
    "in the email never seems to work. Can someone please look into this?"
)
system_estimate = estimate_tokens(system_prompt)
ticket_estimate = estimate_tokens(ticket_text)
total_estimate = system_estimate + ticket_estimate
print(f"System prompt: ~{system_estimate} tokens")
print(f"Ticket text:   ~{ticket_estimate} tokens")
print(f"Total input:   ~{total_estimate} tokens")

  • Lines 1–7: We define estimate_tokens, which estimates token usage by dividing the number of characters by four. Adding three before integer division rounds the result up and prevents partial groups of four characters from being discarded.

  • Lines 10–13: We define the fixed system prompt containing the classification task, permitted categories, and required output format.

  • Lines 14–17: We define the variable support-ticket text that will be classified.

  • Lines 19–21: We estimate the ...