Route Events with Partitions

Learn about adding concurrency in the reduce function using a partition layer.

Problems with Flow.reduce

As we know, Enum.reduce/3 works within a single process. With Flow.reduce/3, we have many processes where each one continuously receives batches of items to work on. This enables Flow to balance the workload and handle back-pressure. However, items that should count together can end up being consumed by more than one process, so we get duplicates when all results are combined in the end. This figure illustrates the problem:

Get hands-on with 1200+ tech skills courses.