Search⌘ K
AI Features

Detailed Design of Kafka

Explore Kafka's detailed design focused on its core components: producers, brokers, and consumers. Learn how Kafka handles real-time data streaming by managing message production, storage, and consumption with scalability and fault tolerance through partitions, leader brokers, and consumer groups.

Kafka is a messaging system that includes a sender that publishes messages (data) not specifically to a receiver but assigns a type to them and a receiver that subscribes to a certain type of messages. Kafka also has a broker that facilitates both the sender and receiver publishing and subscribing to messages.

Kafka’s Architecture

Kafka is composed of three main components, which are described as follows.

Producers

A producer is responsible for creating new messages. They contain the following components:

  • Producer record: A record is kept for each message created in the producer record. It is composed of the topic that’s supposed to get the message, a key or partition, and a value.

  • Serializer: The first thing a producer does on a message is that it converts the key and value of a message into byte arrays (this process is sometimes called the marshaling of data).

  • Partitioner: After being serialized, the message goes to the partitioner, which returns the partitions of a specific topic to which the message should be assigned to. Following are a few ways a partitioner can ...