Implement prepare_messages/2
Explore how to implement the prepare_messages/2 callback in Broadway to preload and enrich messages before processing. Understand how batching database queries improves performance, and learn best practices for managing preloaded data alongside main business logic in concurrent pipelines.
When it comes to preloading data, the prepare_messages/2 callback is here to help.
The prepare_messages/2 callback function
It’s very common to have to do some work in bulk when receiving messages. For example, we may want to fetch or preload information from the database. Rather than doing this in handle_message/3 for each message, we can run a single database query from prepare_messages/2.
The prepare_messages/2 callback runs before handle_message/3 and receives a list of messages. This means you can iterate over the list, look at the data, and update it with more information. It also ...