Model Landscape

Shipping an LLM feature usually starts with constraints that have nothing to do with clever prompting. If customer data cannot leave a region, a 200 ms p95p95 stands for the 95th percentile. It is a latency metric used to measure performance across requests. A 200 ms p95 latency budget means that 95% of all requests must receive a response in 200 milliseconds or faster, allowing only 5% of requests to exceed that time limit. latency budget exists, and the product needs tool use with structured outputs, the main decision becomes where inference runs and what integration surface is available.

Best model is not a stable target because provider policies, latency, and available interfaces change faster than most product cycles. This lesson maps common model options into decision dimensions we can defend in design reviews.

Two operating envelopes, different ownership

Comparing the architecture stacks makes the division of responsibilities clear: Hosted APIs offload infrastructure entirely, while open weights require managing the full stack yourself.

With a hosted API, the application sends a request to an external inference service and receives tokens back over the network. The provider operates the inference server, handles GPU scheduling, patches drivers, and scales capacity, so application code mostly owns prompt construction, request routing, retries, and response validation.

With open weights, the same prompt becomes an input to an inference server we operate. The model weights are deployed on our GPUs, giving us control over runtime settings, batching, and when to roll out updates. However, this control comes with operational responsibilities like capacity planning, queue behavior under load, security patching, and incident response.

Open-weight vs closed-weight in observable terms

The boundary is operational, not philosophical.

Open-weight models run from weight files we can host, so we can inspect artifacts like model configuration, choose the inference engine, and constrain where data flows by constraining where the server runs.

Closed-weight models run on provider infrastructure, so we cannot inspect weights or change the inference stack, but we often get a stable endpoint, managed scaling, and provider-side safety and abuse controls.

Two workflows highlight the difference without diving into deployment details.

• Hosted API workflow: Application code formats input, sends it to an endpoint, then validates the output before it reaches users or tools.

• Open-weight workflow: Application code calls an internal inference service, and the team also monitors GPU utilization, queue depth, and model server health because those now affect user-visible latency and error rates.

Comparing families by their surfaces

Once the ownership model is clear, the next question is what surface each family exposes to engineers. Compare families like GPT, Claude, Gemini, and Llama by reading what their docs commit to in interfaces and policies, not by assuming a permanent ranking from a single eval.

The table below contrasts what changes across families and across open versus closed weights along dimensions that affect engineering work.

The comparison should steer how we verify claims. For example, when docs mention context length, confirm it in an integration test by sending a near-limit prompt and checking for truncation or errors, because the failure mode is visible as missing earlier instructions. When docs describe tool calling or structured outputs, validate by round-tripping a schema and rejecting responses that do not parse, because quality is only useful if it is machine-checkable.

Near-neighbours that cause wrong decisions

Open-weight is not the same as open-source. A model can publish weights while restricting commercial use, restricting redistribution, or limiting modification, which matters for product licensing and downstream distribution.

Closed-weight does not mean no control. Teams still control prompts, tool schemas, retrieval inputs, model routing, temperature, post-processing, and fallbacks, and those levers often dominate user outcomes more than swapping one family for another.

A model choice does not fix quality by itself. If the system sends ambiguous instructions, retrieves irrelevant context, or skips output validation, a different model often produces a different failure, not a reliable solution.

A decision framework we can reuse

Start the selection in the order that prevents late-stage rewrites. Governance questions first because they can force architecture.

• Governance and data handling: Decide whether inputs can leave our environment, whether logs can store prompts, and what audit trail is required.

• Latency and context needs: Set p95 and p99 targets, then test with realistic concurrency and near-limit contexts to observe queueing and truncation behavior.

• Integration surface: Require tool calling, JSON mode, or other structured output constraints, then verify parsing and tool execution safety in tests.

• Cost envelope: Set an upper bound per request and per day, then decide whether that pushes us toward batching, caching, or self-hosting.

Only after those constraints are satisfied should we compare response quality on representative tasks, using the same prompts, the same retrieval, and the same validators.