Go through the different components of the Kafka tool involved in building data pipelines step by step.

Become a Big Data Professional

Apache Kafka is a mainstay platform for building highly scalable data pipelines. This module starts with a review of the fundamentals of Kafka. Moving ahead, we will cover the functioning of the major components of Kafka with practical examples. In the end, we will take a comprehensive look at the internal architecture of Kafka. By the end of this module, we will have the knowledge to create scalable data pipelines using Kafka.

Building Scalable Data Pipelines with Kafka



Whenever we deal with distributed systems, there are five attributes particular to such systems that we must be aware of and understand.

# Reliability
Reliability is a system’s ability to continue to work correctly in spite of faults. A distributed system is usually made of several smaller sub-components that together work to deliver a service. A reliable system can be banked upon to continue to work without degradation of service if a part of the overall system fails. The reliability concept can be extended to include a system’s ability to continue to perform with the expected functionality, tolerate human errors and unexpected use of the system, maintain performance under high data volume load, and, prevent any unauthorized use or abuse of the system when failures do happen.

## Fault vs. failure
The terms “fault” and “failure” are often used interchangeably but mean different things. When a part of a system experiences failure but the system as a whole can still continue to operate reliably, we label the system as **fault-tolerant** or **resilient**. The system can tolerate components deviating from the spec but still function correctly. A failure occurs when the system as a whole fails. No system can be made fault-tolerant from all types of possible faults and can always potentially fail as a whole.

Netflix’s [Chaos Monkey](https://netflix.github.io/chaosmonkey/) is an example of a tool designed to test the resiliency of services in the Netflix ecosystem. The tool randomly terminates service instances to uncover service failures.
 
# Scalability
A system is said to be **scalable** if it can continue to work correctly as the **load** on the system increases. 

## Load
Load can be measured by a variety of metrics, including the number of requests per second received by a web server, the number of **reads from** vs **writes** to a cache, the frequency of a data back-up, the number of supported concurrent users, and more. When considering scalability, we generally question if the system can continue to perform well when one of the load parameters increases.

## Examples
Let’s look at some well-known products and examples of their load parameters:

- **Microsoft Word:** The number of geographically distributed users who can concurrently modify a document in collaboration mode without affecting usability for each user.

- **Facebook:** The number of users able to view a live broadcast from, say, a celebrity. It becomes harder to broadcast a live feed to all users if there are many users viewing a particular live broadcast.

- **Google:** Storage costs for indexing the web as the web grows.

- **Twitter:** Adding a tweet by a popular user instantly to the timeline of all the followers. A tweet by an account with several million followers, if stored in a database, will be fetched millions of times when computing the timelines for each of the account followers and is a potential bottleneck.

## Measuring performance
When the load parameters increase, a scalable system is expected to keep its expected performance. Amazon’s [S3 service level agreement](https://aws.amazon.com/s3/sla/) (SLA), for example,   promises a certain performance benchmark to its users. For instance, the customers are entitled to service credits if S3’s monthly uptime is less than 99.9%.

Let’s consider another example. Instagram has an internal SLA to load a user’s home page in less than X milliseconds; let’s say 5 milliseconds. However, the team can’t guarantee that every home page will load at exactly 5ms. Thus, performances are measured in percentiles. Instagram engineers can set standards such as 99.9% of users will have their home pages loaded in under 10ms, 70% will have their home pages loaded in under 7ms and the median load time would be 5ms. Median implies that half the Instagram users would have their home screens loaded in under 5ms. Theoretically, there could be a very thin minority that sees load times greater than 10ms and due to random events outside the developer's control. Generally, optimizing for very high percentiles, e.g., 99.99th percentile (1 in 10,000) comes with diminishing returns and is usually not worth optimizing for.

## Horizontal vs. vertical scaling
Once load starts to increase on a system, it may be necessary to re-architect the system to handle the new load. There are generally two ways to scale a system demanding more resources:

1. **Vertical scaling**: Add a more expensive or beefier machine than the one on which the current system runs. Consider a MySQL server that runs slowly on the current machine. If we replace it with a new machine with ten times the memory and computing power than the current one, the database server will be able to process a much larger number of queries.

2. **Horizontal scaling**: Horizontal scaling refers to distributing load across several smaller machines. There's a ceiling to vertical scaling as machines can only become so powerful before they run into the limits of physics. This is assuming you don't hit the budget limits of your company in the first place. Scaling horizontally brings complexity with it, especially for stateful services. A distributed database is not a trivial service to maintain when spread across several machines. However, stateless services are much easier to scale horizontally.

In practice, a hybrid approach is usually selected to address scaling issues which uses more powerful machines that also spread load horizontally. Note that there is no one-size-fits-all architecture for distributed applications operating at scale, as each application can have unique load parameters. Access patterns may be different and so too the SLA’s response time requirements.



This lesson discusses the various cornerstones of distributed systems: reliability, scalability, availability, consistency, and maintainability.

Characteristics of Distributed Systems

Reliability