A practitioner's guide to building scalable systems

Building Scalable Systems: Distributed Systems basics to System Design Interviews

We want to give you the tools to compare multiple approaches and pick the one that works for your scenario.

Grokking the Distributed and Scalable Systems

# What is Data Partitioning?
When we launch a new application, just like a single application server, it typically uses a single database server to store data (e.g. you would see database servers like MySQL or Postgres being used). Just like any other Node in a distributed system, the Database server has the following resources that limit the infinite scale

1. Storage Capacity
2. Memory
3. Processing Power
4. Network Bandwidth

Eventually, your database will exceed the maximum capacity available for one or more of the above resources. That's when Data Partitioning comes to the rescue.

<div style="background-color:#81bd69;color:white;padding: 10px; padding-left:30px;">
<i>Database Partitioning</i> is the process of splitting a single large database into multiple smaller databases.
</div>

## What are the benefits of Database Partitioning?

 Partitioning data provides the following benefits


* **Enables Scalability:** Data Partitioning is a big undertaking but once you adapt your application to work with partitioned data, it opens the door for creating scalable applications that can handle millions (or billions) or users/connections/queries, etc.

* **Performance Improvements**: Data Partitioning improves performance and query latency by balancing the load among multiple servers.

* **Maintenance becomes easier:** Maintenance operations e.g. updating the software or hardware can be handled easily as rebooting or updating a single database doesn't mean that the database is not accessible to all users.

* **Improves Availability:** If a single server goes down, it won't cause a service-level downtime.

## Disadvantages of Data Partitioning
Data Partitioning is the next best thing after *vertically scaling* the database server isn't feasible anymore. Before reaching that stage, having a single database server should still be preferred because of the following reasons

* **More servers to manage:** Earlier, you were only responsible for patching and updating a single database server. Now, you are managing multiple servers.

* **Client Applications become complex:** Your database access layer needs to be aware of the partitions and route the client requests to the correct database server that's serving the partition that the client is looking for.


Partitioning a single database into multiple databases increases the complexity of the application.


# What is Data Partitioning?
When we launch a new application, just like a single application server, it typically uses a single database server to store data (e.g. you would see database servers like MySQL or Postgres being used). Just like any other Node in a distributed system, the Database server has the following resources that limit the infinite scale

1. Storage Capacity
2. Memory
3. Processing Power
4. Network Bandwidth

Eventually, your database will exceed the maximum capacity available for one or more of the above resources. That's when Data Partitioning comes to the rescue.

<div>
<i>Database Partitioning</i> is the process of splitting a single large database into multiple smaller databases.
</div>

## What are the benefits of Database Partitioning?

 Partitioning data provides the following benefits


* **Enables Scalability:** Data Partitioning is a big undertaking but once you adapt your application to work with partitioned data, it opens the door for creating scalable applications that can handle millions (or billions) or users/connections/queries, etc.

* **Performance Improvements**: Data Partitioning improves performance and query latency by balancing the load among multiple servers.

* **Maintenance becomes easier:** Maintenance operations e.g. updating the software or hardware can be handled easily as rebooting or updating a single database doesn't mean that the database is not accessible to all users.

* **Improves Availability:** If a single server goes down, it won't cause a service-level downtime.

## Disadvantages of Data Partitioning
Data Partitioning is the next best thing after *vertically scaling* the database server isn't feasible anymore. Before reaching that stage, having a single database server should still be preferred because of the following reasons

* **More servers to manage:** Earlier, you were only responsible for patching and updating a single database server. Now, you are managing multiple servers.

* **Client Applications become complex:** Your database access layer needs to be aware of the partitions and route the client requests to the correct database server that's serving the partition that the client is looking for.


Partitioning a single database into multiple databases increases the complexity of the application.


What and Why

Node in a Distributed System

Performance and Numbers

Fault Tolerance and Availability

Understanding TIME in Distributed Systems

Data Centers for software engineers

Scaling Distributed Systems

Load Balancers

Structured Storage at Scale

Sharding in Storage Systems

SQL vs NO-SQL

Blob Storage at Scale

Caching in Distributed Systems

Queues and Asynchronous Processing

Data Analytics and Batch Processing

Logging and Monitoring

Searching at Scale

Conclusion

Scaling Data Storage using Data Partitioning

What is Data Partitioning?