Detailed Design of Kafka
Understand Kafka's detailed system design, including how producers publish messages, brokers manage data storage and replication, and consumers retrieve data. Learn about Kafka's architecture components and consumer group dynamics to build efficient real-time streaming applications.
We'll cover the following...
Kafka is a messaging system that includes a sender that publishes messages (data) not specifically to a receiver but assigns a type to them and a receiver that subscribes to a certain type of messages. Kafka also has a broker that facilitates both the sender and receiver publishing and subscribing to messages.
Kafka’s Architecture
Kafka is composed of three main components, which are described as follows.
Producers
A producer is responsible for creating new messages. They contain the following components:
Producer record: A record is kept for each message created in the producer record. It is composed of the topic that’s supposed to get the message, a key or partition, and a value.
Serializer: The first thing a producer does on a message is that it converts the key and value of a message into byte arrays (this process is sometimes called the marshaling of data).
Partitioner: After being serialized, the message goes to the partitioner, which returns the partitions of a specific topic to which the message should be assigned to. Following are a few ways a partitioner can operate on messages:
If we specify a key in the producer record, the partitioner uses a hash function on the message key and maps it to a specific partition.
If we specify a partition in the ...