Deployment Surfaces, Voice, Model Choice, and Cost
Understand deployment options for AI agents, including infrastructure and data handling. Explore voice interaction architectures and how they impact system design. Learn model selection criteria balancing quality, speed, and cost. Calculate comprehensive operational expenses accounting for retries, caching, and workloads to prepare AI systems for real-world deployment and interview discussions.
We'll cover the following...
Northstar's finance lead approved a monthly budget of $126 for the operator assistant. The FDE based the estimate on one model call per task, assuming 6,000 cached input tokens and 500 output tokens for each of 18,000 monthly tasks.
Two weeks into the pilot, projected monthly costs exceeded $400, even though the assistant was functioning as designed. The estimate excluded additional model calls for orchestration, retry costs, and applicable tool charges. It also assumed a 100% prompt-cache hit rate.
The model with the lowest cost per call does not necessarily produce the lowest end-to-end deployment cost. This lesson examines the operational decisions that follow a working prototype: where to deploy the agent, how to manage data retention, whether to support voice interactions, which model to use, and how to estimate total operating costs.
These decisions are also relevant to FDE interviews because they require candidates to connect technical choices with operational constraints, customer requirements, and measurable costs.
Where should the agent live?
OpenAI offers three broad homes for an agent, and each moves the interface, the stored data, and the on-call duty to a different owner.
Home | What It Is | Who Runs the Loop and Gets Paged |
Inside ChatGPT | The customer connects its systems to ChatGPT, for example through an MCP server, and teams can package repeatable work as shared Workspace Agents | Largely OpenAI’s product, governed by the customer’s workspace settings |
ChatKit in the customer’s product | OpenAI’s embeddable chat interface, connected to a backend the customer runs | The customer, for the backend and everything behind it |
The customer’s own service | The customer’s application calls an Agents SDK service it operates | The customer, end to end |
Northstar already has an operator console, so the strongest default is an Agents SDK service behind it, with ChatKit as an option if Northstar wants a ready-made chat interface. ChatKit’s recommended setup is the advanced integration, where the customer runs a ChatKit server and connects it to any agent backend. The older hosted path built on Agent Builder is being retired, with shutdown scheduled for November 30, 2026 on OpenAI’s deprecations page, and customers on it need a migration plan: inventory the workflow’s instructions, its tools and its approval points, rebuild it in code, and replay the old test cases against the new version before cutting over.
Running our own service means operating it. Northstar’s console log from the first week showed 24 draft submissions, six client timeouts each followed by a retry, and three duplicate drafts on operators’ screens. The tempting fix treats each timeout as a failure and resubmits with a new request ID. But a timeout only means the client stopped waiting. The server may still be working, or may have ...