Design of Distributed Messaging Queue (Part 1)

So far, we have discussed the requirements and design considerations of a distributed messaging queue. Let’s begin now with the high-level design of a distributed messaging queue.

Distributed messaging queue

Unlike a single-server messaging queue, a distributed messaging queue resides on multiple servers. A distributed messaging queue has its own challenges; however, it resolves the drawbacks of a single-server messaging queue if designed properly.

The following sections will focus on the scalability, availability, and durability issues by introducing a more fault-tolerant architecture of a messaging queue.

High-level design

Before diving deep into the design, let’s assume the following points to make the discussion simple and easy to understand. Further, we will discuss how the following assumptions enable us to eliminate the problems in a single-server solution to messaging queue.

1. Queue data will be replicated using either a primary-secondary or quorum-like system inside a cluster (see Data Replication lesson for details). Our service can use data partitioning if the queue gets too long to fit on a server. We can use a consistent hashing-like scheme for this purpose or may use a key-value store where the key might be the sequence numbers of the messages. In that case, each shard will be appropriately replicated (see Partition lesson for details).

2. We also assume that our system can auto-expand and auto-shrink the resources as per the need to optimally utilize resources.

The following figure demonstrates a high-level design of the distributed messaging queue comprising several components.