Cover the basics and learn how to design and develop distributed systems.

Become a Distributed Systems Professional

Gone are the days when monolithic applications were the norm. Most applications we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems.

In this module, we’ll learn what distributed systems are, and learn about the challenges unique to distributed systems. We’ll cover popular architectural patterns for achieving targets like replication and sharding. We’ll complete the module with a study of two popular distributed systems: Apache Spark and Apache Druid. With the knowledge gained through this module, we’ll be able to design and develop distributed systems. We’ll also develop an understanding of optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

Scaling a system is critical to coping with the increasing load imposed on it. For example, think of our hypothetical My Cool App’s growth—it is now a billion-dollar company.

Initially, when the My Cool App had a few hundred concurrent users, handling the load using $3$ to $5$ nodes with a decent amount of resources was not an issue. Life was simple and beautiful.

Things changed with the fame of the app. More and more people started to use My Cool App and within a short period of time, the system started to see $100k$ concurrent users. This increased load definitely requires scaling.

> **Note**: To scale a system, you need to define the load and performance of your system. Based on the numbers you see, you decide whether to scale your system, and how far you should go with scaling.

Let's discuss load and performance measurement.


# Measuring load

How can we measure load in a distributed system?

The answer depends on our system type. Let’s see a few measurement techniques.

## Queries per second (QPS)

One common method of load measurement is to calculate **queries per second (QPS)**, also known as **requests per second (RPS)**.

This basically means how many queries or requests a node needs to handle each second. For example, say a server receives $10M$ requests per day. Then we calculate the QPS as

> $QPS = \frac{1000000}{24 * 60 * 60} = 11.57$

This is roughly $11.6$ requests per second. So we consider this value the QPS of the node.

Now, this is a very simple measurement. If the requests are uniformly distributed over the period of one day, then this measurement would show the number of requests throughout the day accurately. But that may not be the case for many systems.

It is rather common for distributed systems to see a lot more requests during a certain period of the day compared to other periods. For example, a payment server will generally handle more requests during the day compared to night, because most people do make purchases while they are out and about during the day.

> **Note**: A server may need to handle higher QPS during busy hours. Similarly, it can see higher QPS during weekends or workdays, or on specific holidays. For example, Amazon will likely have more QPS during Christmas or Black Friday.

Bottom line, using QPS and user behavior, we can measure the load of a system. With this insight, it’s now easier to decide when to scale the system.

As an example, if we observe that our existing system with 5 nodes is capable of handling a QPS of $100$ and struggles when it becomes $500$, then we know what to do.

## Read-to-write ratio (r/w)

Apart from QPS, another common way of measuring load in a system is to check the read-to-write ratio (r/w). This is generally used in databases.

> An r/w value of $10$ means for each write, the database serves $10$ read operations.

Well, how does this help?

With this ratio, we can define whether a system is read-heavy or write-heavy.

### Read-heavy systems

This means the value of r/w is higher. In case of a higher load, we are likely to serve more and more reads. 

In read-heavy systems, adding one or more read-replicas to the database cluster will serve the higher count of read-requests.

A read-replica in a database is a node that contains a copy of the original data. Read requests can generally be forwarded to read replicas so that the reads are evenly distributed, and do not overload the node that primarily handles write requests.

### Write-heavy systems

This is indicated by the low r/w value. When the load increases, the system will see a similar amount of writes compared to reads. Or sometimes, more writes than reads.

Scaling a write-heavy system is more complicated compared to a read-heavy system. The general intuition we have is to add more nodes to handle writes. But this requires careful synchronization between the nodes that are handling writes. We cannot just randomly write one request to one node, and the next one to another. We have to follow a strict pattern. 

Concepts related to this are discussed in the next chapter.

---

Apart from the above two, there are some other ways to measure load in a system. For example, data size, response-request size, number of concurrent users connected to a node, etc. Sometimes, this is system-specific. As an owner, you need to build intuition for business use cases, and consider additional metrics to measure load.

## Measuring performance

Similarly, measuring the performance of a system is defined by the type of the system. For example, in a system where we have the request-response pattern, response time is used to measure performance.

> **Note**: Response time is the time witnessed by the client to receive the response for a request.

As an example, if the response time is $200ms$, it means the client receives the response $200ms$ after sending a request to the server. The response time includes the time required to travel across the network in both directions.

But how do we use response time as a metric for performance measurement?

The most obvious way is to use **average response time**. This is basically the arithmetic mean.

> average response time $=$ $\frac{\sum_{i=1}^nt_i}{n}$

- $n$ = total number of requests being considered
- $t_i$ = response time for $i$th request 

The problem with this approach is that it does not show a full picture of response time, because it does not denote how many users experienced some amount of delay. Let’s try a better approach: **percentiles**.

### Percentiles

If we list the response times of $100$ requests to our server and sort them by the response time, we can now tell the median response time of our server i.e. the response time of the middle request in the sorted list.

Say, the median is $2s$. This essentially means half of the requests to our system are served in less than $2s$.

This is called _percentile_. $50%$ of the requests are served in less than $2s$. So we say that the $50th$ percentile (also denoted as $p50$) of our system is $2s$.

It's pretty common for distributed systems to evaluate higher percentiles, like $95th$, $99th$, or $99.9th$.

> If a system has $99th$ percentile as $500ms$, it means out of $100$ requests
sent to the system, $99$ are served in less than $500ms$ time. The $1$ request left may take more than $500ms$, or $1s$, or even more.

For a system, a lower response time for a higher percentile means good performance. Say, if a system has  $99.9th$ percentile as $100ms$, then only $1$ in $1000$ requests will be slower than $100ms$. For most businesses, these sorts of response times are adequate and aren’t a concern. Furthermore, optimizing beyond a certain percentile may not be feasible in terms of costs.

---

# Key takeaways 

- Measuring load and performance will show you the correct direction for scaling out your system.
- As an owner, you have to proactively measure these metrics to ensure that the system has resources according to its needs.

Learn how to measure load and performance in a distributed system.

Measuring Load and Performance