Cover the basics and learn how to design and develop distributed systems.

Become a Distributed Systems Professional

Gone are the days when monolithic applications were the norm. Most applications we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems.

In this module, we’ll learn what distributed systems are, and learn about the challenges unique to distributed systems. We’ll cover popular architectural patterns for achieving targets like replication and sharding. We’ll complete the module with a study of two popular distributed systems: Apache Spark and Apache Druid. With the knowledge gained through this module, we’ll be able to design and develop distributed systems. We’ll also develop an understanding of optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

In this lesson, we will discuss the concept of reliability, the first pillar of fault-tolerant systems.

# What is reliability?

According to M Kleppmann, who put it in simple words in Designing Data Intensive Application, a reliable system is capable of “continuing to work correctly, even when things go wrong.”


Let's explain more.

When you build a distributed system, it makes hardly any sense if your system is not capable of handling the following:

- The system can serve users' expectations, for example, if My Cool App is a photo-sharing app, then users should be able to share photos. If someone uploads a photo and the photo is not shown on their profile, then this leads to a bad user experience.
- If users make mistakes, the system should be able to tolerate the mistake. If a user of My Cool App uploads a video whereas the expectation is only photos, then the system should not break but handle it correctly. For instance, maybe the app will show some error message to the user that the operation is not allowed.
- The system should be performant. If your user opens the feed of My Cool App and waits for 2 minutes for it to load, your users won't be very happy.
- If some malicious user tries to abuse your system, the system should endure it.

As discussed previously, in a distributed system, faults are inevitable. Hardware withers away with time. Networks drop and fluctuate randomly. Power outages happen now and then. When the system is built, scenarios like these have to be kept in mind. A reliable system can operate correctly in times of such failure scenarios. The word correctly is crucial here.
> **Note**: When faults occur, a reliable system is capable of identifying, isolating, and potentially correcting it by itself. If correction is not possible, it should trigger a higher-level recovery mechanism, for example, triggering fallbacks to replacement hardware or software components. If that does not work, it should halt the affected program, and stop the entire system if required. And notify the system owner or developers accordingly.

A reliable system never continues to operate on corrupted data without any recovery effort. If this happens, the system is not reliable.

# What hinders reliability?

In a distributed system, reliability is hindered by different kinds of faults. Let's discuss the two major forms of faults.

## Hardware faults

Do you know how dependable disks are?

>Hard disks are reported as having a mean time to failure (**#key# MTTF: The average time until a component fails. #key#**) of about 10 to 50
years. Thus, on a storage cluster with 10,000 disks, we should expect on average
one disk to die per day.
><div align="right""> <em>Designing Data-Intensive Applications</em>, M Kleppmann </div>

So yeah, as we mentioned earlier in the course, components are failing all the time in a large system. Disks are just one example of this.

There are many hardware components in a single machine. The chance of failures just increases when you have tens of thousands of machines. Apart from the hardware components of the nodes, there is the network that binds the nodes. And networks are also unpredictable.

## Software faults

Machines run server programs to serve user requests. And, as we know, programs tend to have bugs.

A bug in your server software code can be critical or mild. Sometimes it will randomly cause the server program to crash and restart, such as when some null pointer dereferences due to some specific logical flow which is probably rare. Sometimes it will keep the program crashing every single time it tries to restart, maybe due to some misconfiguration in server startup parameters.

Sometimes the bugs are not generic. You won’t encounter them until a very specific set of parameters are passed to a particular function. If you don’t know the specific set of inputs, it may be even impossible to reproduce the bug.

In your system, as the developer, you are probably responsible for a software bug. But sometimes it can be due to some third-party library that you are using. Maybe they have some misplaced condition that caused a set of side effects which affected your system. 

The point is, there will be software faults. And to make your system reliable, you need to handle them.

# Key takeaways

- A distributed system has to be reliable for its users. If a system is not reliable, its users will become increasingly skeptical about it.
- To ensure reliability, faults should be properly detected and handled.
- Both the hardware and software of a system will have faults in them—this is more or less inevitable. We can never expect things to always go right.

Learn about reliability in distributed systems.

Reliability

What is reliability?