Design Considerations in a Distributed Task Scheduler

Queueing

A distributed queue is a major building block used by a scheduler. The simplest scheduling is to push the task into the queue in a first-come, first-served fashion. If there are 10,000 nodes (resource) in a cluster (cloud), the task scheduler will quickly extract tasks from the queue and schedule them on the nodes. But, if all the resources are currently busy, then tasks will need to wait in the queue, and small tasks might need to wait longer. Relying only on this scheduling can affect the reliability of the system, availability of the system, and priority of tasks. There could be cases where we want urgent execution of a task. For example, a task that notifies a user that his/her account was logged in from an unrecognized device. So, we can’t rely only on the first-come, first-serve to schedule tasks. Instead, we categorize the tasks and set the priorities. We have the following three categories for our tasks:

• Tasks that can’t be delayed.
• Tasks that can bear some delay.
• Tasks that need to be executed periodically (for example, every 5 minutes or every hour or every day).