Scheduling
Explore the role of YARN scheduling in managing finite cluster resources. Understand the differences between FIFO, Capacity, and Fair schedulers, including how they allocate resources, support queue management, and handle job fairness and preemption to improve cluster efficiency.
We'll cover the following...
We'll cover the following...
Scheduling
A responsibility of YARN is scheduling user jobs. Scheduling is important because the cluster has finite resources to allocate to users’ jobs based on some policy. Without scheduling, a rogue job can hog the entire cluster and starve other jobs of resources. There’s no silver bullet when it comes to scheduling, as no one policy satisfies all possible usecases. YARN comes with a choice of schedulers and configuration knobs. The three schedulers in YARN are:
- FIFO Scheduler stands for first-in-first-out scheduler. Using this scheduler the jobs are placed in a queue and executed in the order of their submission. Though simple, the FIFO scheduler isn’t suitable for a shared cluster. A large job could deny