Kafka Controller
Explore the role of the Kafka controller in managing cluster membership and leader election. Understand how brokers register with Zookeeper, how the controller maintains cluster stability, and the process of handling broker failures and rejoins. This lesson prepares you to grasp Kafka's internal coordination mechanisms crucial for scalable data pipelines.
We'll cover the following...
If you are unfamiliar with Zookeeper, we would suggest you read the Zookeeper chapter in the appendix first and then come back to this lesson.
Broker membership
So far, we have largely skipped the internal working details of a Kafka cluster and its interactions with consumers and producers. In this chapter, we’ll take a closer look at how the different Kafka components work. We’ll start with the Kafka cluster that consists of several brokers working together. Brokers maintain their membership in a cluster via a unique ID that is set either in the configuration file or automatically generated. Each broker creates an ephemeral node in Zookeeper with its ID under the Zookeeper path /brokers/id. Various Kafka components receive notifications when brokers join or leave the cluster by keeping a watch on the path /brokers/id where brokers create ...