Important Concepts in Product Architecture - IV

Managing retries

Digital applications handle millions of requests per second, yet many fail to complete the request-response cycle. These failures generally surface as specific HTTP status codes, which fall into four categories:

In the first two categories, the client receives no response and must retry. In the last two categories, the client receives an error response and must retry after making changes:

• Request lost: The request is initiated but never reaches the target service, as if it never occurred.

Requests can be lost due to congestion at intermediate routers.

• Response lost: The server processes the request and sends a response, but it never arrives at the client.

• Service unresponsive: The target service cannot process the request (for example, due to a downstream dependency error) and returns an error status with an appropriate message.

• Response with error code: The client receives an error due to a bad request or invalid parameters. The client must address the error before retrying.

In the first two scenarios, the client cannot distinguish which case occurred because no response arrives. In the third scenario, the server returns an error indicating that it cannot process the request. In these three cases, the client’s only option is to retransmit the same request, a process called a retry. The fourth scenario requires the client to fix request errors before retrying.

How do retries lead to issues?

Although rate limiters handle frequent requests within a specified window, issues often arise before reaching the rate-limit threshold:

• Retries from clients that never received a response lead to unnecessary utilization of computational resources at the service.

• Numerous clients retrying simultaneously can congest the network when many requests hit the target service concurrently.

• Useless retries occur when the server’s response will not change (for example, due to a persistently bad request). These waste bandwidth, overburden the server, and cause network congestion.

• Excessive retries hurt the client, too: free APIs may throttle the client’s requests, and paid services waste the API call quota.

How do we manage retries?

To prevent server overload and network congestion, several techniques control retry behavior and discourage retries when they are not needed.

The backoff algorithm

One solution adds jitter to each client’s wait time so that retries do not cluster around the same network failure window. The exponential backoff algorithm uses a multiplication factor to progressively increase the interval between retries:

Here, $time$ represents the delay between requests, $m$ is the multiplication factor, and $c$ is the number of times a specified event (HTTP status code) has occurred. This works well when clients are not retrying simultaneously. But if hundreds of clients retry concurrently, the retries may overshoot available capacity and trigger elastic scalingElastic scaling is the ability to automatically add or remove computer or network resources when needed.. This is the ...