Worker Pool Pattern

Overview of the worker pool pattern

As the name suggests, a pool of workers will distribute work between themselves and send the output to the result channel. The worker pool will receive the job from the jobs channel and distribute the work among themselves.

Let’s create two buffered channels, one to push the jobs and the other to store the results. The workerChannel stores all the available worker channels. When a job arrives, the collector pulls a worker channel from workerChannel and passes the job to it. The worker finishes the work and is added to the pool again for further reuse.

We can use the worker pool pattern to limit the number of concurrent processes going on at one time. Generally, a worker pool is a collection of threads waiting to get tasks assigned to them. Once they finish the assigned task, they make themselves available for the next task.

Let’s see the visual representation of the worker pool pattern: