Evaluation of Distributed Task Scheduler Design

Availability

The first component in our design was a rate limiter that is appropriately replicated and ensures availability. Task submission is done by several nodes. If a node that is submitting a task (send the task to a unique ID generator for storing task ID along with the task attributes into the database) fails, the other nodes take its place. The queue in which we push the task is also distributed in nature, ensuring availability. We always have resources available as we continuously monitor if we need to add or remove resources. Each component in the design is distributed and makes the overall system available.

Durability

We store the tasks in a persistent distributed database and push the tasks into the queue near their execution time. Once a task is submitted, it is in the database until its execution.

Scalability

Task submitter is distributed. We can add more nodes to the cluster to submit an increasing number of tasks. The tasks are then saved into a distributed relational database which is also scalable. The tasks from RDB are then pushed to a distributed queue which can scale with an increasing number of tasks. We can add more queues for different types of tasks. We can add more resources depending on the resource-to-demand ratio.