Back-of-the-Envelope Calculations for Latency

Response time, latency, and why they matter

Modern data-oriented applications continuously process and serve dynamic information to clients over the internet via APIs. At the API design level, we must establish API SLAs that are realistically achievable within our technology and cost budget. For voice calls over the internet, for instance, one-way latency exceeding 100 ms degrades the listener's experience, giving designers a hard threshold to target. If the target is unachievable end-to-end, you must design mitigation strategies.

Major services like Google Search have set high customer expectations; a slow API means an abandoned app. Two questions drive effective experience design:

• How quickly does the API act on requests and return responses?

• How does increasing request volume affect API performance?

Different APIs access different memory types and perform varying operations, producing different latencies. Standard reference numbers from known hardware and network benchmarks underpin the calculations that follow.

Note: A region is a geographic location, a zone is an isolated location within a region, and a data center is the physical instantiation of resources in a zone. A region can have multiple zones, and a zone can have multiple data centers. Intra-zone communication occurs between data centers; inter-zone communication occurs between zones within a region.

Defining the core equation

Sending a request and receiving a response takes time that is composed of two distinct components:

• Latency (network latency): the propagation time of request and response messages between the client and server, excluding processing.

• Processing time: the time the server spends executing queries, computation, and file handling.

API response time spans from request initiation to response receipt, encompassing both network latency and processing time. Although some references use latency and response time interchangeably, they measure entirely different intervals. The governing equation is:

Minimizing response time requires lowering latency, processing time, or both. Latency depends on factors such as the distance between communicating machines and intermediate network components (caches, proxy servers), as discussed here. A ping measures endpoint latency, but a data-retrieval request incurs a substantially higher response time than a simple ping.

Note: Appropriate response time is use-case dependent. Generally, an API is effective with an average response time between 0.1 and 1 second. A 500 ms response time would be unacceptable for a multiplayer online game.

Factors affecting response time

When a client does not know the server's IP address, the request first resolves via DNS, then performs a TCP handshake, followed by an SSL/TLS handshake to establish a secure channel. Only then does the HTTP request reach the destination server for processing and database interaction. Response time breaks down into these segments:

• DNS lookup: time to resolve the IP address via the DNS server.

• TCP handshake: time to establish the initial client-server connection.

• SSL/TLS handshake: time to create a secure communication channel.

• Transfer start: time to acquire the first byte of response data, including both $RTT_{get/post}$ (round trip time of GET/POST) and server-side processing time.

• Download: time for the client to fetch the full response data.

Note: DNS lookup, TCP handshake, and SSL/TLS handshake times are collectively called base time for future reference.

Latency is calculated by excluding the server's processing time using the following general equation:

$RTT_{get/post}$ includes the message propagation time to and from the server.

With the response time equation established, the next step is to estimate each component: processing time first, then network latency.

Estimating processing time

Recall the core equation:

Processing time is the time a server takes to prepare a response, and it is a critical factor in overall response time. The server may interact with databases for query execution, which can involve file handling. Processing time includes the round trip from the API gateway to downstream services, request execution, and response preparation.

No rule of thumb yields exact processing time; it depends on services, components, and hardware and software technologies. Processing typically involves analyzing a query and fetching data from server memory or a database. Three primary factors determine processing time:

• The type of request

• The application server's time to handle a request

• Database query execution time

Processing time depends on the machine's specifications. We consider a typical AWS server whose specifications are defined below:

Request processing estimation

Server request handling time depends on whether the request is CPU-bound or memory-bound:

• CPU-bound: The CPU is the limiting factor.

• Memory-bound: Memory is the limiting factor.

Assuming each CPU-bound request takes 200 ms and each memory-bound request takes 50 ms, requests per second (RPS) are calculated as follows:

The following terms are used in this calculation:

• $RPS_{CPU}$: CPU-bound requests per second

• $Num_{CPU}$: Number of hardware threads (CPU threads)

• $Task_{time}$ ...