Model Choice, Reasoning Effort, and Cost
Evaluate the complexities of model selection, reasoning effort, and cost estimation when deploying AI models. Recognize that the cheapest model per call may not provide the lowest overall cost due to retries, tool usage, and other operational factors. Assess model performance against quality benchmarks and latency requirements. Calculate costs accurately by accounting for different token rates, caching, and their impact on overall expenditure. Document model selection decisions, justify budget adjustments, and maintain supporting evidence to ensure informed and cost-effective deployment.
We'll cover the following...
Northstar’s first budget for the operator draft came to $126 a month. The real figure was closer to $401, because the task makes two model calls, retries about one in ten, and pays for a tool, and the estimate counted only the first call.
The cheapest model per call is rarely the cheapest deployment, so the model decision has two halves. First, pick the configuration that clears a quality bar on the customer’s own work at an acceptable latency. Then cost the whole task, every call, retry, and tool, under realistic traffic and cache assumptions. The operator draft goes through both below, with its authorization and review rules unchanged.
Which model fits the task?
OpenAI’s current lineup spans a range of capability and price, per million tokens. gpt-6.1-sol balances speed, cost, and intelligence, with near-flagship performance for complex work, at $2 input and $10 output. gpt-6.1-terra is fast and efficient for visual reasoning, precise computer use, and clear, natural writing, at $1 input and $5 output. gpt-6-luna is the fastest and cheapest, built for focused, high-volume work such as classification, extraction, and routing, at $0.10 input and $0.50 output.
All three take a 1,050,000-token context window. The tiers differ in supported reasoning settings. All three support Batch and Flex at half the standard rates. Start from the tier that fits the task’s difficulty, then let your own evaluation decide.
Northstar compares two text configurations on 40 read-only operator drafts. A draft is correct when it follows the authorized source, names the source and its version, and makes no unsupported claims. The bar is at least 38 correct out of 40, with no critical unsupported claims. The fast candidate gets 36 correct at a median latency of 1.2 seconds. The harder reasoning candidate gets 39 correct, with no critical unsupported claims, at a median of 3.5 seconds.
Make the call before reading on.
The harder candidate clears the bar, so it’s the provisional choice for this draft task. Investigate its one failure and check with operators whether 3.5 seconds is acceptable. The fast candidate misses the bar, and its ...