This device is not compatible.


Creating Data Pipeline in Elixir using Kafka

In this project, you will learn to connect a Kafka server with a consumer and producer using Elixir code. In the end, you will stream the producer's CPU percentage to the consumer and log in the console on the consumer side.

Creating Data Pipeline in Elixir using Kafka

You will learn to:

Configure a message queue producer and consumer in Elixir

Start Zookeeper and Kafka

Implement console-based streaming between producers and consumers

Stream data through Kafka


Management of Message Queues

Message Broker Architecture

Data Pipeline Engineering


Basic understanding of Elixir

Basic understanding of Kafka




Project Description

In this project, you’ll use a popular message streaming service known as Kafka with an Elixir project. You’ll use a wrapper to interface between Kafka and our Elixir code. The name of the wrapper library is Kaffe.

First, you’ll set up the Kafka message queue server and then build a simple Elixir application and use Kaffe to connect it to the running Kafka server.

Before you start the project, keep the following terms in mind:

  • consumer: A process that receives the messages.
  • producer: A process that sends the messages.
  • topic: A means to group messages that allows producers and consumers to subscribe only to the messages they want to receive.

Project Tasks


Start Zookeeper and Kafka

Task 0: Getting Started

Task 1: Start Zookeeper

Task 2: Start Kafka


Create the Producer

Task 3: Create the Producer Application

Task 4: Configure the Producer

Task 5: Produce Messages

Task 6: Run Producer


Create Consumer

Task 7: Create the Consumer Application

Task 8: Configure the Consumer

Task 9: Consume Messages

Task 10: Call the Consumer

Task 11: Run Consumer