Design of Distributed Task Scheduler

We identified requirements for a distributed task scheduler in the previous lesson. In this lesson, we will map those requirements to a design.

Components

We can consider scheduling at many levels. It could be scheduling that is done internally by an organization to run their tasks on their own cluster of machines, and they have to find ample resources and need to decide which task to run first. On the other hand, It could be scheduling that a cloud provider uses to schedule tasks coming from multiple clients. Cloud providers need to decide which task to run first and which clients to handle first to provide appropriate isolation between different tenants. So, in general, the big components of the system that we have are:

• Clients that initiate task execution.
• Resources on which the task actually executes.
• A scheduler between clients and resources that decides which task should get the resource first.