Search⌘ K
AI Features
Home
Courses
System Design Deep Dive: Real-World Distributed Systems

Evaluation of Kafka

Understand how Kafka achieves high throughput and efficient data streaming by comparing its producer and consumer performance against ActiveMQ and RabbitMQ. Learn about Kafka's scalability, batching, storage efficiency, and design principles that enable real-time, reliable data processing in distributed systems.

We'll cover the following...

Kafka promised to be efficient in collecting data from multiple producers in parallel, retaining data, and delivering it to multiple consumers simultaneously. Moreover, it promised to deliver loads of data in real time. Let's go through some pieces of evidence as to how Kafka provides these functionalities by comparing the performance of Kafka with Apache ActiveMQ (a popular open-source implementation of Java Message Service (JMS)) and RabbitMQ (a messaging system known for its performance).

All the computational results and time spentMany of the performance numbers and graphs in this lesson are inspired from the paper: Kreps, Jay, Neha Narkhede, and Jun Rao. "Kafka: A distributed messaging system for log processing." In Proceedings of the NetDB, vol. 11, pp. 1-7. 2011. on them that is stated in the text below are done on two Linux machines, both of which have eight 2GHz cores, 16 GB of memory, and 6 disks with RAID 10. Both the Linux machines are connected through a 1 GB network link. One is deployed as a broker, and the other performs the function of both the producer and consumer interchangeably. Though such an experimental setup might seem minuscule, Kafka can extract amazing throughput from this setup. Since Kafka is horizontally scalable, it will not be a stretch to extrapolate these numbers for a larger setup (for example, for back-of-the-envelope calculations).

Performance

...