Partition Rebalancer
Explore how Kafka manages consumer group membership and partition assignments through the partition rebalance mechanism. Understand the role of the group coordinator and consumer leader during rebalances, how partition ownership changes to maintain availability, and the impact on consumer state. This lesson helps you grasp the fundamental concepts that support scalability and fault tolerance in Kafka consumer groups.
We'll cover the following...
Membership of consumers in a consumer group is coordinated by a designated Kafka broker referred to as the group coordinator. It receives heartbeats from consumers to confirm that they are alive and healthy. When the group coordinator doesn’t hear from a consumer for a configurable period of time, it assumes the particular consumer has crashed and triggers a partition rebalance. The latest version of Kafka has a dedicated heartbeat thread that sends heartbeats to the group coordinator periodically. ...