A practitioner's guide to building scalable systems

Building Scalable Systems: Distributed Systems basics to System Design Interviews

We want to give you the tools to compare multiple approaches and pick the one that works for your scenario.

Grokking the Distributed and Scalable Systems

# Why does it matter?
A client wants the server to complete a certain task e.g.

* Upload a Photo to Instagram
* Place an Order at Amazon

For each of these tasks, the client and server will most likely exchange multiple requests and responses. 

The discussion of Stateful vs. Stateless services is only relevant if the client and server are exchanging more than one message to complete a certain task (which is the case most of the time)

# What is State?
When a client starts a session with the server, it might require multiple requests and responses to complete that transaction, e.g., when you are ordering on Amazon, you might add a few menu items to your cart and then click "buy." Every time you are adding an item into the cart, the server is noting it down somewhere. That data about the items in your order is called **state**.

## What is a Stateless application
An application that doesn't save any data between requests associated with a specific session is called a stateless application, e.g., a website that shows you the weather of a particular zip code. Every time, you change the zip code, it goes to the server and fetches the weather for the given zip code without having to rely on what requests were sent before.

## What is a stateful application
The application that saves any "data" or "state" between multiple requests associated with a session is called a stateful application, e.g., Amazon is a stateful application that saves cart across multiple requests.

# Horizontally Scaling a Stateless Application
_Remember that in horizontal scaling, we add a pool of commodity servers to process more requests concurrently._

A stateless application can scale relatively seamlessly because you don't have to worry about sending the request to the same server that has saved it's the earlier state. You can keep adding more servers to the server pool without any problems.



# Horizontally Scaling a Stateful Application


But we might have use cases where it becomes necessary to have maintained state and still horizontally scale the application. Amazon's e-commerce website is a stateful distributed application that needs to scale and maintain state as well.

When an application has to maintain state across requests, and each request might hit a different server, you cannot save state in the server. Instead, there are three ways to manage state in distributed applications.

1. Cookies
2. Shared Data Store
3. Load Balancer with Sticky Sessions


_We will look at both Cookies and Shared Data Store now and the look at the Sticky Sessions using Load Balancers after we've looked at the Load Balancers_ 


## 1. Session data in Cookies
Using a cookie to save session data across multiple requests in the simplest way to persist session data. Let's take a look at it.

### What is a Cookie?
A cookie is a piece of data that the client passes to the server (browser in most cases) with every request in the session except the first one.

When the server receives a request without a session cookie (_first request_), it creates a session cookie for that session and associates the cookie with that session by setting the appropriate HTTP header in the response.  

The client caches that cookie in a file and includes that file in all subsequent requests to the server by setting the appropriate HTTP request header.

### Pros of using Cookies for State Management
Cookies simplify the state management for an application where a pool of servers is processing requests. Every server can look at the HTTP header of the request and extract the cookie. e.g., Amazon can update the cookie with new items in the cart, and when the customer clicks Buy, it can extract all the items from the cookie and place the order. 

Cookies also mean that the servers don't have to do anything extra for session management.

### Cons of using Cookies for State Management
Cookies force servers and browsers to exchange all the information with every request which would work for small amounts of data, e.g., Item IDs.

However, what if you were writing a long email with images and saving its draft before sending it out. If the server were packing all the email text/image data in the response cookie, the client would be downloading MBs of data and then, with every request, uploading the data again that it had already uploaded before. That would create quite a mess. Hence cookies don't work well where the data is larger than a few KBs.

## 2. Session Data in Shared Data Store
We've found a problem with cookies if the data is exceeding a few KBs. That is where server-side shared storage comes into play.

Instead of storing the data itself in the cookie, we create a session identifier (_session_id_) when the first request comes and store that session identifier in the cookie. 

The data is stored in a shared data store like Memcached or Redis. Typically, a low latency key-value store works best for such cases. You would use the session_id as they key and store the data against that key.

The requests can go to any server, and the server will extract the session_id (or issue a new one) and then starting storing/retrieving the data from the shared storage.

## 3. Sticky Session using the Load Balancer
_The only reason I put the Stateless vs. Stateful applications after discussing Load Balancers is that I wanted you to understand L4 and L7 load balancers before discussing this strategy._

If there's an absolute need for storing the state on the server itself, both L4 and L7 Load Balancers can support sticky sessions. Here's how

1. When a request is received for the first time, the load balancer can select the server for that request using the techniques like Round Robin or Least Number of connections, etc. 

2. However, once the server is selected for the session, all the subsequent request would go to the same server.

3. L4 routers can remember the server affinity based on the Source IP and Port of the client.

4. L7 Load Balancers can store the session id in the cookie as well and use that session id to identify the server.

# Final Thoughts: Stateless vs. Stateful Applications

1. As discussed earlier, Stateless applications are easier to scale. Aim for creating Stateless applications as much as possible.
2. If the State is small, the best way is to use cookies. 
3. However, in cases like REST APIs, there is no notion of cookies and the only option to maintain state is to use shared state.
4. Use sticky sessions with Load Balancer only as a last resort. One big downside of sticky sessions is that if the server crashes, the state is lost and the request would have to be canceled and restarted from scratch. In the case of cookies or shared data store, any server can continue serving the request and it wouldn't need to be canceled.



What and Why

Node in a Distributed System

Performance and Numbers

Fault Tolerance and Availability

Understanding TIME in Distributed Systems

Data Centers for software engineers

Scaling Distributed Systems

Load Balancers

Structured Storage at Scale

Sharding in Storage Systems

SQL vs NO-SQL

Blob Storage at Scale

Caching in Distributed Systems

Queues and Asynchronous Processing

Data Analytics and Batch Processing

Logging and Monitoring

Searching at Scale

Conclusion

Stateful vs. Stateless Applications and Session Management

Why does it matter?

What is State?

What is a Stateless application

What is a stateful application