Search⌘ K
AI Features
Home
Courses
Concurrent Data Processing in Elixir

Revisiting the Producer

Explore how to manage producer demand in Elixir's GenStage by implementing handle_demand and handle_cast callbacks. Understand how to expose APIs for dispatching events, create producer-consumer pipelines, and improve scraper performance using concurrency. This lesson helps you build effective data processing flows with GenStage.

We'll cover the following...

Update consumer demand

Now, for our scraper project, let’s lower the demand to something more suitable:

C++
#scraper/lib/page_consumer.ex
def init(initial_state) do
  Logger.info("PageConsumer init")
  sub_opts = [{PageProducer, min_demand: 0, max_demand: 3}]
  {:consumer, initial_state, subscribe_to: sub_opts}
end

With min_demand set to 0 and max_demand set to 3, we ensure that the consumer takes at least one event when available, up to 3 at a time. However, our producer still has to supply the consumers with events, and right now, it doesn’t do anything. Let’s fix this.

Going back to PageProducer, we have implemented the handle_demand/2 callback to respond to demand and produce events. Remember that ...