Stream Processing with Redis Streams

Redis streams

As we mentioned earlier, a Redis stream is a powerful data structure that combines the capabilities of Redis Pub/Sub and list. It offers reliable messaging, durability for replaying messages, consumer groups for load balancing, and pending entry lists for monitoring. Although Redis streams also follow the producer-consumer pattern (like Redis list), its core design is based on an append-only log data structure. This a key differentiator between Redis streams and the other data structures we’ve looked at.

With Redis streams, producers can add records to a Redis stream using the XAdd method, and consumers can subscribe to new items arriving to the stream with the XRead method. It supports range queries like the XRange method and, thanks to consumer groups, a group of application instances can distribute the processing load using the XReadGroup method, and it’s possible to monitor its state as well with the XPending method.

While the methods XAdd, XRange, etc. are generic commands, there are specialized versions (like XReadGroup), which are specific to the Redis streams consumer groups feature. We’ll begin by exploring the generic operations that can be used using the go-redis client; then, we’ll move on consumer groups in the subsequent sections.

The XAdd method

The XAdd method is the only way to add messages to a Redis stream. It appends a key-value entry to the specified stream and returns the ID of the entry that was added. In the example below, we add five key-value entries to test-stream. By default, Redis autogenerates an ID if one is not passed in the call to XAdd.