Designing pub-sub

Using distributed messaging queues

A topic is a sequence of events. When a writer writes a message into the system, it is added to the end of the related topic, and the previous messages are kept the way they were, just like a queue. The readers read from the topic they have subscribed to. Many writers can write into many topics, and many readers can read from many topics. We can use the distributed messaging queues for topics.

Each topic will be a distributed messaging queue. A writer will write their messages to that queue. The message from the queue is deleted once the message is read. So for each reader, we will define a separate queue so that we can push the messages to that queue and the reader can read it from there.

The queues ensure strict ordering, remove the message once consumed by the reader, and is distributed. Using the distributed messaging queues makes our design simple. But the huge number of queues needed is a major concern. If we have millions of subscribers for thousands of topics, defining and maintaining millions of queues is ...