Estimating Inference Capacity and Fleet Size

Previously, we devised a formula to estimate model training time and applied it to inference. Building on that foundation, this lesson performs quick BOTECs for the resources required to deploy an LLM at scale. We cover the following:

• Storage estimation:

  • Model storage

  • User profile data

  • User interaction data

  • Indexing storage

• Inference server and fleet estimation

• GPU cost estimation

• Network bandwidth estimation

With these calculations, we aim to estimate the end-to-end resource footprint for deploying LLMs at scale. Let’s start with resource estimation fundamentals.

Resource estimation

Resource estimation ensures the infrastructure can handle the expected workload efficiently while remaining cost-effective. It determines the compute, storage, and network requirements for scalability, reliability, and performance. Accurate estimation prevents over-provisioning, which inflates costs, and under-provisioning, which causes failures or poor user experience.

Storage requirements

Storage covers the model itself, user profile data, and user interaction data. Production systems may also store metadata, context, logs, prompt templates, and configuration files. We focus on the most essential components.

Let’s start with estimating the model’s storage.

Model storage

The storage for a model depends on its parameter count and the precision used to store each parameter. Data precision can be FP64 (64-bit)It is often known as a double-precision floating point format. This format uses 64 bits to represent numerical values, offering high precision and a wide range of values. It is primarily used in scientific computations and applications requiring extreme accuracy., FP32 (32-bit)It is a single-precision floating point format widely used in machine learning and graphics; this format balances precision and computational efficiency. It is sufficient for most deep learning tasks and is often the default format., FP16 (16-bit)Also known as half-precision, this format reduces memory usage and speeds up computation by representing values with lower precision. It is commonly used to train and infer machine learning models, especially on specialized hardware like GPUs., or quantized (8-bit)This format represents values using 8 bits, significantly reducing memory and computation requirements. While sacrificing some precision, it effectively deploys machine learning models in resource-constrained environments like mobile devices.. This precision shows the size of one parameter.

At FP16 (2 bytes per parameter), a 3-billion-parameter model needs:

Higher precision improves accuracy but increases size and can slow inference. Quantization trades some accuracy for a smaller footprint.

User data and interaction data

Assume each user’s profile takes 10 KB and each user makes 10 interactions per day, with each interaction stored at 2 KB:

Indexing storage

Fast retrieval of interaction data requires an index. Assuming a 25% overhead:

The model and profile data are static, so they are excluded from the daily indexing calculation. The total daily storage is:

Over a 30-day month, this is $2.5 \text{ TB} \times 30 = 75 \text{ TB/month}$ ...