Design of a Pub-Sub System
Define a scalable pub–sub architecture using partitioned topics and dedicated brokers instead of a single queue. Explain how cluster managers coordinate replication and how consumer managers track offsets to support reliable delivery semantics.
We'll cover the following...
First design
Producers write to topics, and consumers subscribe to topics to read messages. As new messages are appended to the end, we can use distributed messaging queues.
The system requires the following components:
Topic queue: A distributed messaging queue where producers write their messages.
Database: A relational database stores subscription details. For example, we need to store which consumer has subscribed to which topic so we can deliver the messages they want. We use a relational model to maintain structured data and ensure integrity.
Message director: Reads messages from the topic queue, fetches subscriber details from the database, and forwards messages to the appropriate consumer queues.
Consumer queue: A dedicated distributed queue for each consumer. Messages are copied here from the topic queue for consumption.
Subscriber: Handles subscription requests and adds entries to the database.
When a consumer subscribes, the subscription metadata is persisted to the system store. The message dispatcher reads from the topic, resolves the list of subscribed consumers, and appends the message to each consumer’s queue. Consumers read messages from their assigned queues.
Note: We use failover services for the message director and ...