Commits and Offsets
Explore how Kafka consumers manage offsets to keep track of their reading position in topic partitions. Understand the concept of commits, their impact on data processing continuity, and issues like duplicate or missed records during consumer rebalances. This lesson prepares you to configure offset commits effectively for reliable data consumption.
We'll cover the following...
We have read so far that Kafka consumers poll topic partitions for records using the poll() method. The method returns some number of records for the consumer to process. However, if a rebalance occurs and the partition gets assigned to a different consumer within the consumer group, then the newly assigned consumer must know where the previous consumer stopped in order to resume reading records from ...