Task Distribution Patterns

Although delegating costly tasks to multiple local processes is an effective approach, it can’t be scaled beyond the boundaries of a single machine, so in this section, we’re going to see how it’s possible to use a similar pattern in a distributed architecture, using remote workers located anywhere in a network.

The idea is to have a messaging pattern that allows us to spread tasks across multiple machines. These tasks might be individual chunks of work or pieces of a bigger task split using a divide-and-conquer approach.

If we look at the logical architecture represented in the following illustration, we should be able to recognize a familiar pattern:

As we can see in the illustration above, the Publish/Subscribe pattern is not suitable for this type of application, as we absolutely don’t want a task to be received by multiple workers. What we need instead, is a message distribution pattern similar to a load balancer that dispatches each message to a different consumer (also called a ...