Static Batching
Explore how static batching works in Broadway within Elixir data ingestion pipelines. Learn to assign messages to different batchers, handle batches concurrently, and implement custom logic for specific batch groups. This lesson helps you manage multiple batchers to optimize parallel processing and tailor data handling for different event types.
We'll cover the following...
Each message returned by the handle_message/3 callback must belong to a batcher. If one hasn’t been assigned, messages go to the :default batcher. This is the group we’ve already defined in start_link/1. Therefore, everything works, and we don’t need to do anything.
Static batching
Unlike :processors, the :batchers configuration supports multiple groups. This means we can have more than one batcher doing the work. We can use this to create batchers per event type. For example, we can have a :cinema batcher and a :musical batcher for cinema ...