Estimating Training Compute and Cost

One of the most critical steps when designing and implementing generative AI (GenAI) systems is resource estimation. Whether you're training a new model or deploying an inference system for millions of users, understanding the required resources (computational power, storage, and bandwidth) can save significant time and cost. However, detailed calculations can be time-consuming and require access to precise technical details. This is where back-of-the-envelope calculations (BOTECs) come into play.

Back-of-the-envelope calculations (BOTECs) are quick, rough estimations that provide a high-level understanding of what a system might require regarding resources. These calculations aren’t about pinpoint accuracy but getting a reliable ballpark figure to guide decision-making. For instance, we might want to estimate how many GPUs we’ll need before training a large language model or how long the training will take. Similarly, for deployment, we can use BOTECs to predict the number of inference servers required to handle user requests.

In this lesson, we’ll explore the basics of BOTECs for training and deploying GenAI systems, starting with the training time and resource estimation.

Why do BOTECs matter in GenAI?

BOTECs help GenAI engineers make informed resource and performance decisions before detailed measurements or tooling are available. Because GenAI systems operate at large scales, even small estimation errors can lead to significant waste in GPU usage, training time, storage, or serving costs. BOTECs also play an important role in technical interviews, where candidates are expected to break down complex system requirements and reason through estimates such as GPU capacity, training time, and storage needs. Developing this skill helps engineers make practical decisions quickly and communicate their reasoning with confidence.

In this chapter, we will focus on three types of BOTECs:

1. Estimating training compute and cost

2. Estimating training compute and cost

3. Token math and unit economics

Let’s start with the model training estimations to devise some mathematical formulations:

Training estimations

In the training phase of building a generative AI system using a large model like the Llama model, we want to determine how long it would take to train it on a certain amount of data. The Llama 2-70B model took around 1.7 million GPU hours using NVIDIA A100 GPUs. Calculating this beforehand allows us to plan resources accordingly and build systems in a realistic time frame. For example, using one GPU to train Llama 2-70B would take ~195 years, while using a cluster of 10,000 GPUs would take around one week. There is also the cost aspect.

As we know, training any model usually means we are passing our data through the model and updating its internal weights. Both these processes require mathematical calculations like multiplying or adding certain values. To estimate the time it takes to train a model, we need the answers to these two questions:

1. How many calculations will be performed?

2. How many calculations can be done in a certain time (e.g., one second)?

The calculations we refer to here are FLOPS (floating point operations). A floating point is simply a number with a decimal point. Without diving into the specifics of a model’s architecture, we cannot know how many calculations will be performed during one forward/backward pass, so we need to make some informed guesses.

Let’s look at an example of how we can estimate these FLOPS:

The network is configured with two input nodes, three hidden layer nodes, and one output node.

The total number of parameters in the network are:

For one feedforward pass through the network, we will need the following calculations:

Where $b_1$ is the bias value of the first layer. We can see that the first few calculations require $4+4+4$ ...