A practitioner's guide to building scalable systems

Building Scalable Systems: Distributed Systems basics to System Design Interviews

We want to give you the tools to compare multiple approaches and pick the one that works for your scenario.

Grokking the Distributed and Scalable Systems

# Let's debug a Process Crash
<div style="background-color:#FFFFA5;padding: 10px; padding-left:30px; font-style: italic;">

__By the way:__

If someone hasn't told you already, I have news for you. There is usually no Debugger in a Distributed System when you are investigating an issue. At most, you'll have Log Files and system metrics (if you are lucky).
</div>

Consider you are debugging a problem that caused downtime in your service as one of the processes crashed. 

You've identified that the issue occurred during communication between **Node-A** and **Node-B**. Processes on both Nodes were creating log files.

Here's how logging worked:

1. For every log write, the process notes the current clock time as the timestamp.
2. It writes the log entry in the log file with the timestamp. 

_Let's look at both log files._

Here's the log file for Node B (we're looking at it first because it's more straightforward).

<div style="background-color:#A569BD;color:white;padding: 10px; padding-left:30px;">

```
NODE-B
10:06:00: Received Message from Node A
10:06:01: Sent Response to Node A
```

</div>

and here's the log file for Node-A

<div style="background-color:#A569BD;color:white;padding: 10px; padding-left:30px;">

```
NODE-A
10:05:00: Sent Message to Node B
10:04:00: Received Response from Node B.
10:04:01: This request took... 

** PROCESS CRASHED **
```
</div>

# What happened here?
If you look at the timestamps on both log files, it seems like Node-A sent a message to Node-B which received it after a full minute, then immediately sent a response which was received 60 seconds earlier than the time when the message was sent.

This can't be true so what happened here. 

A few things actually

1. The time on Node-A and Node-B is out of sync by almost 60 seconds. This can happen due to various reasons which we will be looking in detail in this chapter.
2. More importantly, someone changed the clock time of Node-A while the request was in progress. This rendered the log file useless.
3. Also, Node-A crashed while trying to calculate the time spent in the request - presumably because the computed time-spent was negative.

<div style="background-color:#FFFFA5;padding: 10px; padding-left:30px; font-style: italic;">

__By the way:__

1. Funny thing is that many times the processes crash not due to a product bug but while logging.

2. More ironic are the cases where the process crashes when trying to log an error. Because errors are rare and it's difficult to test for all error paths, that crash-prone code can go undetected for years. 

3. It sucks, even more, when you have to fix the logging, deploy a new build and then wait for reoccurrence of the original bug to debug it.

Been there, done that. Yes, I've been guilty of all 3 of the above.


</div>


Alright, back to debugging our crash in Node-A. 

Someone changed the clock time on Node-A while the request was in progress. We will see in a little bit who could be the culprit (_hint: it's usually NTP_). 

_But for now, remember a few things._

# Takeaways from our little Debugging exercise

Here are a few things that I would like you to remember from this exercise.

## 1. Computer Clocks are unreliable

Computer clocks in a Node are unreliable as the clocks can drift and the time can change abruptly. Any operation relying on the system time is hence unreliable and can show weird behaviors e.g. 

1. It's almost impossible to create the TRUE sequence of events if your logging library is using the system time to timestamp events especially when multiple threads are writing to the same log file. You can't reliably answer who did what at what time.

2. If you have timeout requests, imagine what will happen if the time changes when the request was in progress. Would you be able to timeout the request correctly if the time elapsed is negative? 

## 2.  We want the Clock time to change

It's by design (_again NTP - can't wait to get to that but wait for now_). We actually expect and want the Nodes to change the time as soon as it goes out of sync. We just have to know about it and use other ways (more on it soon) that can provide a more reliable way of counting time elapsed and for calculating timestamps for logging.

## 3. Time on different nodes can be very different
If you are relying on clocks of two different nodes for some operations, don't do that. There are ways to detect a partial order between operations on two different nodes (we'll get to that towards the end of this chapter) but there's no way to know the absolute order of operations between multiple nodes. We can it "partial-ordering" and the notion of "happened-before" but again we're getting ahead of ourselves here.

---

_In the following lessons, we will look at how computer clocks work, what is NTP and how the notion of the time works in a distributed system. Let's get started._

# Let's debug a Process Crash
<div>

__By the way:__

If someone hasn't told you already, I have news for you. There is usually no Debugger in a Distributed System when you are investigating an issue. At most, you'll have Log Files and system metrics (if you are lucky).
</div>

Consider you are debugging a problem that caused downtime in your service as one of the processes crashed. 

You've identified that the issue occurred during communication between **Node-A** and **Node-B**. Processes on both Nodes were creating log files.

Here's how logging worked:

1. For every log write, the process notes the current clock time as the timestamp.
2. It writes the log entry in the log file with the timestamp. 

_Let's look at both log files._

Here's the log file for Node B (we're looking at it first because it's more straightforward).

<div>

```
NODE-B
10:06:00: Received Message from Node A
10:06:01: Sent Response to Node A
```

</div>

and here's the log file for Node-A

<div>

```
NODE-A
10:05:00: Sent Message to Node B
10:04:00: Received Response from Node B.
10:04:01: This request took... 

** PROCESS CRASHED **
```
</div>

# What happened here?
If you look at the timestamps on both log files, it seems like Node-A sent a message to Node-B which received it after a full minute, then immediately sent a response which was received 60 seconds earlier than the time when the message was sent.

This can't be true so what happened here. 

A few things actually

1. The time on Node-A and Node-B is out of sync by almost 60 seconds. This can happen due to various reasons which we will be looking in detail in this chapter.
2. More importantly, someone changed the clock time of Node-A while the request was in progress. This rendered the log file useless.
3. Also, Node-A crashed while trying to calculate the time spent in the request - presumably because the computed time-spent was negative.

<div>

__By the way:__

1. Funny thing is that many times the processes crash not due to a product bug but while logging.

2. More ironic are the cases where the process crashes when trying to log an error. Because errors are rare and it's difficult to test for all error paths, that crash-prone code can go undetected for years. 

3. It sucks, even more, when you have to fix the logging, deploy a new build and then wait for reoccurrence of the original bug to debug it.

Been there, done that. Yes, I've been guilty of all 3 of the above.


</div>


Alright, back to debugging our crash in Node-A. 

Someone changed the clock time on Node-A while the request was in progress. We will see in a little bit who could be the culprit (_hint: it's usually NTP_). 

_But for now, remember a few things._

# Takeaways from our little Debugging exercise

Here are a few things that I would like you to remember from this exercise.

## 1. Computer Clocks are unreliable

Computer clocks in a Node are unreliable as the clocks can drift and the time can change abruptly. Any operation relying on the system time is hence unreliable and can show weird behaviors e.g. 

1. It's almost impossible to create the TRUE sequence of events if your logging library is using the system time to timestamp events especially when multiple threads are writing to the same log file. You can't reliably answer who did what at what time.

2. If you have timeout requests, imagine what will happen if the time changes when the request was in progress. Would you be able to timeout the request correctly if the time elapsed is negative? 

## 2.  We want the Clock time to change

It's by design (_again NTP - can't wait to get to that but wait for now_). We actually expect and want the Nodes to change the time as soon as it goes out of sync. We just have to know about it and use other ways (more on it soon) that can provide a more reliable way of counting time elapsed and for calculating timestamps for logging.

## 3. Time on different nodes can be very different
If you are relying on clocks of two different nodes for some operations, don't do that. There are ways to detect a partial order between operations on two different nodes (we'll get to that towards the end of this chapter) but there's no way to know the absolute order of operations between multiple nodes. We can it "partial-ordering" and the notion of "happened-before" but again we're getting ahead of ourselves here.

---

_In the following lessons, we will look at how computer clocks work, what is NTP and how the notion of the time works in a distributed system. Let's get started._

Time is tricky even within a single node. It's even more tricky in Distributed Systems. Let's see why

What and Why

Node in a Distributed System

Performance and Numbers

Fault Tolerance and Availability

Understanding TIME in Distributed Systems

Data Centers for software engineers

Scaling Distributed Systems

Load Balancers

Structured Storage at Scale

Sharding in Storage Systems

SQL vs NO-SQL

Blob Storage at Scale

Caching in Distributed Systems

Queues and Asynchronous Processing

Data Analytics and Batch Processing

Logging and Monitoring

Searching at Scale

Conclusion

Did this Node Time-Travel into the Past?

Let’s debug a Process Crash