Scheduling

This lesson explains the various scheduling algorithms that YARN uses when scheduling jobs.

Scheduling

A responsibility of YARN is scheduling user jobs. Scheduling is important because the cluster has finite resources to allocate to users’ jobs based on some policy. Without scheduling, a rogue job can hog the entire cluster and starve other jobs of resources. There’s no silver bullet when it comes to scheduling, as no one policy satisfies all possible usecases. YARN comes with a choice of schedulers and configuration knobs. The three schedulers in YARN are:

  • FIFO Scheduler stands for first-in-first-out scheduler. Using this scheduler the jobs are placed in a queue and executed in the order of their submission. Though simple, the FIFO scheduler isn’t suitable for a shared cluster. A large job could deny smaller jobs from running.

Get hands-on with 1200+ tech skills courses.