This device is not compatible.

Visualizing Live Streaming Data with Elixir, Kafka, and Chart.js

PROJECT

Visualizing Live Streaming Data with Elixir, Kafka, and Chart.js

This project focuses on using Phoenix Sockets and Channels to display a real-time chart. We'll stream your CPU utilization, with the memory usage, through the producer and plot it in real-time on the consumer side.

You will learn to:

Develop real-time web applications

Use WebSockets in Elixir

Use Channels to create a soft real-time functionality

Plot data dynamically using Chart.js

Skills

Message Broker Architecture

Web Development

Distributed Message-based Architecture

Prerequisites

Basic understanding of WebSockets and Channels

Basic understanding of Kafka

Basic understanding of JavaScript

Basic understanding of Elixir

Technologies

Kafka

Elixir

Phoenix

Chart.js

JavaScript

Project Description

In this project, we’ll use the popular message streaming service Kafka with two Elixir applications. The two primary processes in Kafka are the producer, which sends messages, and the consumer, which receives those messages. We’ll use the Kaffe wrapper library to interface Kafka and your Elixir code.

We’ll first start the Kafka and Zookeeper servers. Then, we’ll develop the producer application. This application will send the CPU utilization percentage and the memory used by all the processes to the Kafka server.

After configuring and implementing the producer, we’ll modify an already-given basic consumer application to receive those messages. Then, we’ll plot the data in a real-time chart using the Chart.js library.

Project Tasks

Start Zookeeper and Kafka

Task 0: Start Zookeeper

Task 1: Start the Kafka Server

Configure the Producer

Task 2: Add the Application

Task 3: Produce Messages

Configure the Consumer

Task 4: Create a Socket

Task 5: Create a Channel

Task 6: Receive Messages

Task 7: Modify the Template

Plot Data in Real-Time

Task 8: Create a Chart

Task 9: Prevent Duplicate Labels

Task 10: Distinguish the Data

Task 11: Create New Labels

Task 12: Add New Points to the Chart

Task 13: Add the Event Handler

Run the Applications

Task 14: Run the Producer and Consumer

Congratulations!

Hear what others have to say

Join 1.4 million developers working at companies like

"Another great hands on project to apply your knowledge learned. Thank you Educative ❤️"

Atabek BEKENOV

Senior Software Engineer

"Super excited to learn E-commerce website for my own startup venture. Thanks for your great learning platform."

Pradip Pariyar

Senior Software Engineer

"This was an excellent lesson. I learned a lot working through the process. I enjoyed it so much that I rebuilt it my AWS account to see how hard it would be to deploy to a production environment."

Renzo Scriber

Senior Software Engineer

"It was my first proper data engineering project and it was amazing."

Vasiliki Nikolaidi

Senior Software Engineer

"It's a fantastic way to do hands-on practice; I enjoy this way of learning."

Juan Carlos Valerio Arrieta

Senior Software Engineer

Relevant Courses

Use the following content to review prerequisites or explore specific concepts in detail.