Handling Rebalances
Explore how Kafka consumers manage partition rebalancing through the ConsumerRebalanceListener interface. Learn to implement onPartitionsRevoked and onPartitionsAssigned methods for tasks like committing offsets, cache management, and setting read positions. Understand strategies to avoid message duplication and handle offsets using external databases during rebalances.
We'll cover the following...
A consumer may want to do cleanup work before a partition it owns is reassigned during a rebalance. They may also want to prepare before being assigned a new partition. Both use cases are addressed by the interface ConsumerRebalanceListener. The interface offers two methods that can be implemented by classes to insert custom logic just before a partition is revoked or assigned.
-
onPartitionsRevoked: This method is invoked after a consumer stops consuming events but before a rebalancing takes effect. -
onPartitionsAssigned: This method is invoked after partitions have been assigned to a consumer but before it starts consuming messages.
Example use of onPartitionsRevoked
We can implement the interface and pass an instance of the implementing class at the time of invoking the subscribe() call of the consumer object. The code widget below ...