Search⌘ K
AI Features
Home
Courses
Mastering Apache Kafka

The Kafka Producer API

Explore how to use the Kafka Producer API to send messages to topics reliably and efficiently. Learn the differences between synchronous and asynchronous producers, configure key settings, and apply practical Java examples to build high-performance Kafka producers.

We'll cover the following...

Introduction to Kafka producers

In Apache Kafka, producers are responsible for publishing messages (events) to Kafka topics. These messages can represent anything—from user clicks and system logs to sensor readings or financial transactions.

In Apache Kafka, producers are responsible for publishing messages (events) to Kafka topics. These messages can represent anything—from user clicks and system logs to sensor readings or financial transactions.

At a high level, sending data to Kafka using the Producer API involves just a few steps:

  1. Configure the Kafka producer

  2. Create a KafkaProducer instance

  3. Send messages to a Kafka topic

While this sounds simple, how you send messages has a big impact on:

  • Performance

  • Reliability

  • Latency

Kafka supports two primary producer patterns:

  • Synchronous producers

  • Asynchronous producers

Let’s explore both and understand when to use each.

Producer configuration basics

Before sending messages, we configure the producer using a java.util.Properties object.

Common configuration properties include:

  • bootstrap.servers – Kafka broker address

  • key.serializer – Converts keys to bytes

  • value.serializer – Converts values to bytes

  • linger.ms – Controls batching delay

These settings determine how efficiently and reliably messages are delivered to Kafka.

Tip: Producer configuration plays a major role in throughput and durability—especially in high-volume systems.

How synchronous producers work

In synchronous mode, the producer:

  • Sends a message to Kafka

  • Waits for a response from the broker

  • Continues only after receiving acknowledgment

This is done by calling .get() on the Future<RecordMetadata> returned by the send() method.

When to use it

Synchronous producers are useful when:

  • Every message must be confirmed

  • Failures must be handled immediately

  • Throughput is less important than correctness

The downside is lower performance, since each send blocks execution.

Synchronous producer example

Go through the example below and click the “Run” button to execute the code.

package com.example;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Future;

import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

public class KafkaProducerExample {

  public static void main(String[] args) throws Exception {
    String topicName = "sync-test-topic";

    System.out.println("Executing synchronous Kafka producer");

    Properties properties = new Properties();
    properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "1");

    Producer<String, String> producer = new KafkaProducer<>(properties);

    for (int i = 0; i < 5; i++) {

      String key = "sync-key-" + i;
      String value = "sync-value-" + i;

      ProducerRecord<String, String> record = new ProducerRecord<>(topicName, key, value);

      Future<RecordMetadata> response = producer.send(record);
      response.get();

      System.out.println("Produced data: " + key + "=" + value);

      Thread.sleep(1000);
    }

    producer.flush();
    producer.close();

    System.out.println("End of program");
  }
}
Sending messages to Kafka synchronously
Code explanation

Let’s walk through important parts of the code and also understand the output of the program above:

  • Lines 3–9: We import packages from the standard library

  • Lines 11–16: We import packages from the Kafka library.

  • Line 21: We set the name of the Kafka topic to send messages to. In this case, it’s sync-test-topic.

  • Lines 25–29: We create a new Properties object to specify the configuration for the Kafka Producer.

    • We set the bootstrap servers property to localhost:9092, which specifies the location of the Kafka brokers.

    • We set the key and value serializer properties to org.apache.kafka.common.serialization.StringSerializer.

    • We set the linger.ms property to 1, which specifies how long the Kafka Producer waits for additional messages to batch together before sending them to the Kafka broker.

  • Line 31: We create a new KafkaProducer instance using the configuration properties.

  • Lines 33–46: We start a loop that sends five messages to the Kafka topic. For each iteration, perform the following:

    • Generate a key and value for the message.

    • Create a new ProducerRecord instance with the topic name, key, and value.

    • Send the record (key-value pair) to the Kafka broker using the send method. It returns a Future object representing the acknowledgment of the message from the broker.

    • Call the get method on the Future object to block until the message has been successfully sent to the broker.

    • Print the key, and the value is successfully sent to the broker.

    • Wait for one second before sending the next message.

  • Line 49: We close the KafkaProducer instance.

How asynchronous producers work

In asynchronous mode, the producer:

  • Sends messages without waiting for Kafka’s response

  • Uses a callback function to handle success or failure

  • Continues sending messages immediately

This approach allows Kafka to batch messages efficiently and maximize throughput.

When to use it

Asynchronous producers are ideal when:

  • Performance and scalability matter

  • Occasional retries are acceptable

  • Systems process large volumes of events

Most production Kafka pipelines use asynchronous producers.

Asynchronous producer example

Go through the example below and click the “Run” button to execute the code.

package com.example;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Future;

import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

public class KafkaProducerExample {

  public static void main(String[] args) throws Exception {
    String topicName = "async-test-topic";

    System.out.println("Executing asynchronous Kafka producer");

    Properties properties = new Properties();
    properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "1");

    Producer<String, String> producer = new KafkaProducer<>(properties);

    CountDownLatch counter = new CountDownLatch(5);

    for (int i = 0; i < 5; i++) {

      String key = "async-key-" + i;
      String value = "async-value-" + i;

      ProducerRecord<String, String> record = new ProducerRecord<>(topicName, key, value);

      producer.send(record, new Callback() {

        @Override
        public void onCompletion(RecordMetadata metadata, Exception e) {
          if (e != null) {
            System.out.println("Error sending data " + key + e.getMessage());
          } else {
            System.out.println("Produced data: " + key + "=" + value);
          }
          counter.countDown();
        }

      });

      Thread.sleep(1000);
    }

    counter.await();

    producer.flush();
    producer.close();

    System.out.println("End of program");
  }
}
Sending messages to Kafka asynchronously
Code explanation

Let’s walk through important parts of the code and also understand the output of the program above:

  • Lines 3–9: We import packages from the standard library.

  • Lines 11–16: We import packages from the Kafka library.

  • Line 21: We set the name of the Kafka topic to send messages to. In this case, it’s async-test-topic.

  • Lines 25–29: We create a new Properties object to specify the configuration for the Kafka Producer.

    • We set the bootstrap servers property to localhost:9092, which specifies the location of the Kafka brokers.

    • We set the key and value serializer properties to org.apache.kafka.common.serialization.StringSerializer.

    • We set the linger.ms property to 1, which specifies how long the Kafka Producer waits for additional messages to batch together before sending them to the Kafka broker.

  • Line 31: We create a new KafkaProducer instance using the configuration properties.

  • Line 33: We create a CountDownLatch object with a count of 5 . This is used to wait until all messages have been sent before closing the producer.

  • Lines 35–57: We start a loop that asynchronously sends 5 messages to the Kafka topic. For each iteration, we perform the following:

    • Generate a key and value for the message.

    • Create a new ProducerRecord instance with the topic name, key, and value.

    • Use the send method with a Callback object to send the record to the Kafka broker. The Callback object handles the acknowledgment of the message from the broker.

    • In the Callback object’s onCompletion method, check if an exception occurred while sending the message. If an exception occured, print an error message. Otherwise, print a message indicating the key and value of the message that was sent. Then, decrement the count of the CountDownLatch object.

    • Wait for one second before sending the next message.

  • Line 59: We call the await method on the CountDownLatch object to wait until all messages have been sent and acknowledged by the broker.

  • Line 62: We close the KafkaProducer instance.

Note: A CountDownLatch has been used in this program for demonstration purposes so that we can see all the messages being sent successfully. Without this, the program would end without us being able to see the output. In a production application, this is typically not used when we want to send messages to Kafka asynchronously.

In this lesson, we covered synchronous and asynchronous Kafka producers and their differences, along with practical examples of how to use these with the Kafka Java client.