This course immerses the learner into the design of systems and makes them ready for any new design problems.

sys-design.tar.gz

basic

This course provides an alternate perspective to system design where we learn about building blocs first and then use them to build bigger, more capable systems.

Copy-System Design-The Missing Pieces

# Requirements

There are tasks in the system that have elastic latency requirements. The examples are: 

* Delivering answers
* Sending notifications
* Generating digest emails

Such tasks can tolerate some delay or they take substantial time and are not a good candidate to synchronously call from a web or API task.

# Asynchronous queueing system
Tasks can be put in a managed queue and can be delivered to the consumers.  Following is the architectural diagram.



![](/api/collection/10370001/6050282698178560/page/6308844024365056/image/6392753424695296?page_type=collection_lesson)


## Kafka
While there are many options available on the cloud, if we are cost-sensitive, Kafka is a good choice. Once the message is accepted to the Kafka system, it provides necessary reliability and scalability.

## Kafka consumer service

This service batch retrieves data from Kafka and puts them in RAM to the server. Similarly, this service stores the results back to Kafka from memory over time.

## Execution with Qmessages

Qmessages get tasks from the KCS service, get them executed using an appropriate system, and put the result back to KCS. For horizontal scalability, we can add more servers to the Qmessage group.

Asynchronously Handling Billions of Tasks Per Day

Requirements