Sending Messages

There are three ways in which messages can be sent by producers:

• Fire and forget: Message is sent to Kafka with no effort made to verify if the message was received successfully by the Kafka broker. In most instances, the messages will be received by Kafka, since the producer has retries built-in for failures. However, this method is still prone to losing messages when retries exhaust.

• Synchronous: Message is sent and the future object returned has its get() method invoked to see if the Kafka broker received the message successfully.

• Asynchronous: Messages are sent and a callback is specified, which is invoked when a response is received from the Kafka broker. In this method, the producer doesn’t wait to receive a response from Kafka before continuing to send other messages.

In the example from the previous lesson, we sent the messages synchronously, since the call to future.get() doesn’t return until the Kafka broker responds with success or failure. However, this approach doesn’t scale when we have millions of messages to be sent. If the network latency for a round trip from the producer to the Kafka cluster is 10ms, the time it takes to synchronously send millions of messages will be unbearable for any serious distributed system. The solution, then, is to send messages asynchronously and specify a callback that is invoked when a response from the Kafka cluster is received. Let’s rewrite our previous example as an asynchronous producer.