Asynchronously Handling Billions of Tasks Per Day

Requirements

There are tasks in the system that have elastic latency requirements. The examples are:

• Delivering answers
• Sending notifications
• Generating digest emails

Such tasks can tolerate some delay or they take substantial time and are not a good candidate to synchronously call from a web or API task.

Asynchronous queueing system

Tasks can be put in a managed queue and can be delivered to the consumers. Following is the architectural diagram.

Kafka

While there are many options available on the cloud, if we are cost-sensitive, Kafka is a good choice. Once the message is accepted to the Kafka system, it provides necessary reliability and scalability.

Kafka consumer service

This service batch retrieves data from Kafka and puts them in RAM to the server. Similarly, this service stores the results back to Kafka from memory over time.

Execution with Qmessages

Qmessages get tasks from the KCS service, get them executed using an appropriate system, and put the result back to KCS. For horizontal scalability, we can add more servers to the Qmessage group.