High-level Design of Kafka
Explore the high-level design of Kafka to understand its core components such as producers, brokers, and consumers. Learn about messages, topics, partitions, and how Kafka achieves scalability and reliability through batching and streaming APIs for real-time data processing.
We'll cover the following...
Data flow
Let's start with the basic units of Kafka and build up the concepts of the components in which they are saved.
Message
The basic unit of data in Kafka is called a message. It can be thought of as a row in a CSV file or a record in a database. Messages contain a payload of bytes. It can also have a payload of metadata, referred to as a key. Keys can be hashed to write messages to certain partitions in a
Batch
Messages are exported to Kafka in batches to increase its throughput. A batch is a set of messages that exist in the same partition of a topic.
Batching messages eliminates the need for each message to require a full ...