Implement Batching and handle_batch/4
Explore how to implement batching and the handle_batch/4 callback in Broadway to group and process data events concurrently. Understand batch configuration and improve performance by bulk inserting records in Elixir data pipelines.
We'll cover the following...
Implement batching
The tickets application currently handles several types of events and creates records in the database for each. We use batching to group the incoming events by type and insert them all together.
To start using batching, we need to add a configuration to enable it. We can then implement the handle_batch/4 callback—the last callback on our list.
Just like handle_message/3, handle_batch/4 is special, and all code within the callback runs concurrently in a separate batch processor. Batching works by configuring one or more batchers. Each batcher starts a single batch processor by default, which runs the code in ...