Wrapping Up

This chapter wasn’t the easiest, so congratulations on completing it! Although slightly intimidating at first, GenStage is a brilliant tool that gives Elixir an advantage over other programming languages. Of course, as Elixir’s creator José Valim has pointed out many times, none of it would be possible without Erlang’s rock-solid OTP serving as a foundation.

Back-pressure

This chapter looked at back-pressure and its problems when building and scaling data processing pipelines.

GenStage

We also learned about GenStage and how stages enable us to use back-pressure to build data processing pipelines easily.

ConsumerSupervisor

We saw how we could scale our system to handle increased workloads by leveraging concurrency with ConsumerSupervisor. Multistage pipelines can help model and tackle potential challenges when working on more complex tasks. We can adapt these techniques to build resilient and scalable systems that perform well under increased workloads.

Different configurations of GenStage

We saw that GenStage is very versatile and can be configured in many different ways to solve an endless variety of problems.

Advanced features of GenStage

Although we covered a lot of ground, we have only seen a fraction of what’s possible. There are more advanced features in GenStage that didn’t make it in this chapter, such as the ability to switch to the :manual subscription mode in stages of the :consumer and :producer_consumer type. This gives us the flexibility to change demand at runtime, rather than hard-coding max_demand and min_demand at compile time.

These and other features are well documented online. You can refer to GenStage’s official documentation and explore what’s available further.

The feasibility of GenStage

Working with GenStage could be challenging. Everything must be configured and set up correctly for our data processing pipeline to work. To achieve the desired concurrency level, we may have to create more than a few types of stages. This often makes GenStage a less appealing choice for more common tasks and problems, especially in teams less experienced. The good news is that José Valim and the team have worked on two more libraries—Flow and Broadway—built on top of GenStage. These are designed to solve specific problems with less configuration and setup.

---