Go through the different components of the Kafka tool involved in building data pipelines step by step.

Become a Big Data Professional

Apache Kafka is a mainstay platform for building highly scalable data pipelines. This module starts with a review of the fundamentals of Kafka. Moving ahead, we will cover the functioning of the major components of Kafka with practical examples. In the end, we will take a comprehensive look at the internal architecture of Kafka. By the end of this module, we will have the knowledge to create scalable data pipelines using Kafka.

Building Scalable Data Pipelines with Kafka

Kafka, founded by Jay Kreps, Neha Narkhede, and Jun Rao, emerged on the Big Data scene in January of 2009. It has since enjoyed increasing popularity and use, especially in the enterprise space. Any Big Data department will almost certainly have a team using and maintaining Kafka or a wrapper service over Kafka for the rest of the company to use. Kafka is written in Java and Scala and has its roots in LinkedIn, where it was originally developed before being open-sourced. The original use case for Kafka was to track a user’s actions on the LinkedIn website. These actions served as inputs for an array of applications on the backend such as machine learning systems, search optimizers, and report generators which all play an important role in enriching the user experience.

This lesson introduces the reader to Kafka and gives an overview of the technology. 

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Appendix

Conclusion

Introduction