High-level Design of Kafka

Data flow

Let's start with the basic units of Kafka and build up the concepts of the components in which they are saved.

Message

The basic unit of data in Kafka is called a message. It can be thought of as a row in a CSV file or a record in a database. Messages contain a payload of bytes. It can also have a payload of metadata, referred to as a key. Keys can be hashed to write messages to certain partitions in a topic A stream of particular type of messages. in an organized way. This assures us that the messages with similar keys exist in the same partition.

Batch

Messages are exported to Kafka in batches to increase its throughput. A batch is a set of messages that exist in the same partition of a topic.