HomeCoursesBuilding Scalable Data Pipelines with Kafka

AI-powered learning

Save

Building Scalable Data Pipelines with Kafka

Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

4.4

62 Lessons

Join 2.9 million developers at

LEARNING OBJECTIVES

Learn the theory behind Kafka
Interact with a Kafka cluster running in the browser-terminal

Learning Roadmap

62 Lessons

Basics

Step through the fundamentals of Kafka, distributed systems, messaging patterns, and core components.

Introduction

Characteristics of Distributed Systems

Kafka Producer

Unpack the core of Kafka Producers, message sending methods, configurations, and serialization techniques.

Producer

Sending Messages

Producer Configurations

Producer Serialization

Kafka Consumer

9 Lessons

Go hands-on with Kafka consumers, configurations, offsets, and partition rebalancing techniques.

Kafka Internals

7 Lessons

Break down complex ideas of Kafka's replication, controller, request processing, and reliability.

Conclusion

Compare Kafka's scalability, throughput, and real-time processing with other messaging systems.

Kafka vs Other Messaging Systems

Appendix

3 Lessons

Activate Zookeeper insights, practical API use, and common distributed system solutions.

Reference: Replication

14 Lessons

Master the principles of replica management, leader-based and leaderless replication strategies, and conflict resolution methods.

Reference: Partitioning

4 Lessons

Learn how to use partitioning strategies to enhance scalability and optimize data pipelines.

Reference: Transactions

9 Lessons

Discover the logic behind managing data transactions, isolation levels, and concurrent write challenges.

10.

Reference: Issues in Distributed Systems

4 Lessons

Examine the challenges in developing and maintaining distributed systems, including networking, time synchronization, and handling failures.

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Developed by MAANG Engineers

ABOUT THIS COURSE

If you’re interested in Big Data, then Apache Kafka is a must-know tool. What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

ABOUT THE AUTHOR

DataJek

A bay area tech outfit, throwing lots of good ideas on the wall to see what sticks!

Learn more about DataJek

Trusted by 2.9 million developers working at companies

These are high-quality courses. Trust me the price is worth it for the content quality. Educative came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks

Anthony Walker

@_webarchitect_

Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!

Evan Dunbar

ML Engineer

You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it.

Software Developer

Carlos Matias La Borde

I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site

Souvik Kundu

Front-end Developer

Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content.

Vinay Krishnaiah

Software Developer