HomeCoursesConcurrent Data Processing in Elixir

Intermediate

11h 10min

Updated 1 month ago

Concurrent Data Processing in Elixir

Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Join 2.7 million developers at

Overview

Content

Reviews

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance. This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines, and run concurrent processes with Elixir. By the end of the course, you’ll have a wide range of new techniques to simplify your product, improve code performance, and make your application more resilient to increased workloads.

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in s...Show More

WHAT YOU'LL LEARN

Understand how concurrency works in Elixir

Learn about Elixir supervisors and GenServers

Learn to use data processing pipelines

Learn to use the Flow module in Elixir

Learn to set up data ingestion pipelines

Understand how concurrency works in Elixir

Content

78 Lessons7 Quizzes

Getting Started

2 Lessons

Get familiar with concurrent data processing in Elixir and its prerequisites.

What Does This Course Cover?

Course Prerequisites and Intended Audience

Easy Concurrency with the Task Module

16 Lessons

Unpack the core of concurrent programming in Elixir with the Task module and supervision.

Introduction to Concurrency

An Introduction to the Task Module

Create the Playground

Start Tasks and Retrieve Results

Synchronous vs. Asynchronous Programs

Retrieve a Task’s Results

Use the await and yield Functions to Retrieve Results

Stop a Task and Async Task

Manage a Series of Tasks

Parameters of the async_stream Function

Link Processes

Meet the Supervisor

Use Task.Supervisor

Understand the “Let It Crash” Philosophy

Wrapping Up

Quiz: Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

19 Lessons

Work your way through GenServer callbacks, process management, and optimizing concurrency in Elixir.

Long-running Processes and Starting GenServer

GenServer Callbacks: init

GenServer Callbacks: handle_continue and handle_call

Genserver Callbacks: handle_cast

GenServer Callbacks: handle_info

GenServer Callbacks: terminate

Make Improvements to Our GenServer

The blueprint of our job-processing system

How to Use Anonymous Functions

Introduce DynamicSupervisor

Process Restart Values and Frequency

Implement a Supervisor

Supervisor Strategies

Registry Processes

Limit Concurrency and Inspect the Supervisors

Examine Running Processes and Stop the Supervisors

Wrapping Up

Quiz: Long-running Processes Using GenServer

Create a GenServer and Supervisor From Scratch

Project

Data Processing Pipelines with GenStage

13 Lessons

Find out about GenStage's role in managing data pipelines, back-pressure, and concurrency.

Back-Pressure

Introduce GenStage

The Producer and Consumer

Build the Data Processing Pipeline

Understand Consumer Demand

Revisiting the Producer

Add More Consumers and Buffer Events

Add Concurrency With ConsumerSupervisor

Create Multi-stage Data Pipelines

Scale Up a Stage with Extra Processes

Choose the Right Dispatcher

Wrapping Up

Quiz: Data Processing Pipelines with GenStage

Process Collections with Flow

12 Lessons

Take a closer look at concurrent data processing with Elixir's Flow library for large datasets.

Introduction to the Flow Library

Read Files

Improve Performance with Streams

Run Streams Concurrently and Understand Flows

The reduce Function

Route Events with Partitions

Group and Sort

Windows and Triggers

Get Snapshots from Slow-running or Infinite Flows

Add Flow to a GenStage Pipeline

Wrapping Up

Quiz: Process Collections with Flow

Data Ingestion Pipelines with Broadway

15 Lessons

See how Broadway simplifies data-ingestion pipelines with concurrent processing, batching, and fault tolerance.

Introduction to the Broadway Library

Process Ticketing Events

The Details of Broadway Callbacks

Implement handle_message/3

Implement prepare_messages/2

Implement handle_failed/3

Batch Messages

Implement Batching and handle_batch/4

Static Batching

Use Dynamic Batching

Adjust Batch Size and Timeout

Use GenStage Producers

Configure the Scraping Pipeline

Wrapping Up

Quiz: Data Ingestion Pipelines with Broadway

Concluding the Course

1 Lessons

Learn how to improve your ability to create efficient, scalable Elixir applications.

Course Summary

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Course Author:

The Pragmatic Programmers

Developed by MAANG Engineers

Every Educative resource is designed by our in-house team of ex-MAANG software engineers and PhD computer science educators — subject matter experts who’ve shipped production code at scale and taught the theory behind it. The goal is to get you hands-on with the skills you need to stay ahead in today's constantly evolving tech landscape. No videos, no fluff — just interactive, project-based learning with personalized feedback that adapts to your goals and experience.

Trusted by 2.7 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Software Developer

Carlos Matias La Borde

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI Mock Interviews

Test your skills in a simulated interview setting. Receive personalized feedback based on your performance. Available for Coding Interviews, System Design, and more.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Free Resources

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath