This device is not compatible.

PROJECT


Creating Data Pipeline in Elixir using Kafka

In this project, you will learn to connect a Kafka server with a consumer and producer using Elixir code. In the end, you will stream the producer's CPU percentage to the consumer and log in the console on the consumer side.

Creating Data Pipeline in Elixir using Kafka

You will learn to:

Configure a message queue producer and consumer in Elixir

Start Zookeeper and Kafka

Implement console-based streaming between producers and consumers

Stream data through Kafka

Skills

Management of Message Queues

Message Broker Architecture

Data Pipeline Engineering

Prerequisites

Basic understanding of Elixir

Basic understanding of Kafka

Technologies

Kafka

Elixir

Project Description

In this project, you’ll use a popular message streaming service known as Kafka with an Elixir project. You’ll use a wrapper to interface between Kafka and our Elixir code. The name of the wrapper library is Kaffe.

First, you’ll set up the Kafka message queue server and then build a simple Elixir application and use Kaffe to connect it to the running Kafka server.

Before you start the project, keep the following terms in mind:

  • consumer: A process that receives the messages.
  • producer: A process that sends the messages.
  • topic: A means to group messages that allows producers and consumers to subscribe only to the messages they want to receive.

Project Tasks

1

Start Zookeeper and Kafka

Task 0: Getting Started

Task 1: Start Zookeeper

Task 2: Start Kafka

2

Create the Producer

Task 3: Create the Producer Application

Task 4: Configure the Producer

Task 5: Produce Messages

Task 6: Run Producer

3

Create Consumer

Task 7: Create the Consumer Application

Task 8: Configure the Consumer

Task 9: Consume Messages

Task 10: Call the Consumer

Task 11: Run Consumer

4

Congratulations!

Congratulations!