Inference Infrastructure: Hardware, Economics, and Latency
Explore the critical aspects of inference infrastructure for large language models, focusing on hardware constraints, latency decomposition, and economic trade-offs. Understand GPU and VRAM requirements, quantization impact, and cost comparisons between API-based and self-hosted inference. Learn how to optimize inference pipelines for production reliability and cost efficiency.
In the previous lesson, we hardened our application against misuse and abuse.
The system is now secure enough to operate on the public internet. At this point, however, most LLM applications do not fail due to security incidents. They fail because they are economically unsustainable. In traditional software systems, compute is treated as a commodity.
If a service is slow, we add more CPU cores. If a database grows, we add storage. Costs scale roughly linearly and are usually predictable.
Inference infrastructure for large language models behaves very differently.
Compute is scarce, memory is constrained, and costs can scale non-linearly. A single modern GPU can cost tens of thousands of dollars. Even when rented, it is expensive enough that poor utilization can dominate the entire budget.
Deploying an LLM-backed system without understanding the physics of inference often leads to two outcomes: the service crashes under load due to memory exhaustion, or it quietly drains your budget until the project is shut down.
In this lesson, we will examine how latency actually works in LLM systems, how to calculate hardware requirements deterministically, and how to reason about the economics of API-based inference vs. self-hosting.
This lesson focuses exclusively on inference, not training. Training infrastructure is a fundamentally different problem with different cost and scaling characteristics.
In LLMOps, infrastructure is part of the application’s behavior. Latency, cost, and availability are user-facing features. Decisions regarding hardware, batching, and hosting models significantly impact product reliability and business viability.
Decomposing latency: TTFT vs. TPS
When users ...