Implement Worker Queues with Lists
Learn how to use Redis lists for background job processing.
We'll cover the following
Worker queues
Redis lists support blocking operations (BLPop
, BRPop
, etc.), which makes it possible to wait for a specified time duration if there is no item in the list. As soon as an item is added to the list—for example, by another application using the LPush
method—that item can be retrieved and processed. This allows us to decouple applications and build asynchronous, event-driven solutions. This also known as the worker queue pattern, where the list acts as a queue; the producer applications add items to this queue and the worker applications (consumers) dequeue and process them.
Let’s learn how this works with the help of an application.
Email processor application overview
Sending emails is a common requirement for many websites. This is a perfect use case for leveraging background processing, because we don’t want the user or client to wait while the email is being sent. We’ll go over this a hypothetical solution based on the combination of a REST endpoint and a background job consumer implemented using a Redis list.
This application consists of two parts, outlined below.
Use cases
The following use cases have been implemented in the application:
A REST endpoint to accept emails: A JSON payload representing the email information (recipient and message) is sent to a REST endpoint using HTTP
POST
, and the HTTP handler uses theLPush
method to add this information to a Redis list.A background job to process email requests: A separate consumer application uses the
BRPop
method to retrieve and process email tasks from the job.
Note: The actual email sending process is not implemented as a part of this application.
Code explanation
Let’s walk through the important parts of the code. First, let’s take a look at the REST API that accepts email requests and sends them to the list.
Imports
Here are some of the package imports:
Get hands-on with 1200+ tech skills courses.