This device is not compatible.
PROJECT
Creating Data Pipeline in Elixir using Kafka
In this project, you will learn to connect a Kafka server with a consumer and producer using Elixir code. In the end, you will stream the producer's CPU percentage to the consumer and log in the console on the consumer side.
You will learn to:
Configure a message queue producer and consumer in Elixir
Start Zookeeper and Kafka
Implement console-based streaming between producers and consumers
Stream data through Kafka
Skills
Management of Message Queues
Message Broker Architecture
Data Pipeline Engineering
Prerequisites
Basic understanding of Elixir
Basic understanding of Kafka
Technologies
Kafka
Elixir
Project Description
In this project, you’ll use a popular message streaming service known as Kafka with an Elixir project. You’ll use a wrapper to interface between Kafka and our Elixir code. The name of the wrapper library is Kaffe.
First, you’ll set up the Kafka message queue server and then build a simple Elixir application and use Kaffe to connect it to the running Kafka server.
Before you start the project, keep the following terms in mind:
consumer: A process that receives the messages.producer: A process that sends the messages.topic: A means to group messages that allows producers and consumers to subscribe only to the messages they want to receive.
Project Tasks
1
Start Zookeeper and Kafka
Task 0: Getting Started
Task 1: Start Zookeeper
Task 2: Start Kafka
2
Create the Producer
Task 3: Create the Producer Application
Task 4: Configure the Producer
Task 5: Produce Messages
Task 6: Run Producer
3
Create Consumer
Task 7: Create the Consumer Application
Task 8: Configure the Consumer
Task 9: Consume Messages
Task 10: Call the Consumer
Task 11: Run Consumer
4
Congratulations!
Congratulations!
Relevant Courses
Use the following content to review prerequisites or explore specific concepts in detail.