Design a GPU Training Job Scheduler
Design a multitenant GPU training job scheduler that keeps an 8,000-GPU fleet busy. It uses guaranteed quotas with borrowing of idle capacity, gang scheduling that places each job on well-connected GPUs, and checkpoint-based recovery for preemption and hardware faults, balancing utilization, fairness, and fault tolerance.
Idle GPUs burn cash every minute, but filling them carelessly can make a 512-GPU training run crawl because half its processes communicate across the wrong network boundary. A GPU training scheduler prevents both failures. It acts as the control plane for an accelerator fleet in which jobs last hours or days, request resources in groups, and regularly face hardware faults, preemption, and restarts.
This lesson designs that scheduler from the interview perspective. We cover queueing, placement, preemption, checkpoint recovery, and reporting, with the system framed as a multitenant platform service rather than a single algorithm.
Unlike inference serving, training jobs are usually gang-scheduled, meaning the full process set must start together or not at all. They are also topology-sensitive, so placement directly changes training throughput. The scheduler therefore balances four forces at once: utilization, fairness, priority handling, and fault tolerance.
A useful mental model is an airport gate controller. Empty gates waste money, but assigning a large aircraft to the wrong gate disrupts the whole terminal. The next step is to estimate the fleet and workload before choosing policies.
System requirements
Requirements shape every policy choice, so fix them before sizing the fleet.
Functional requirements
Job submission: Teams submit gang jobs with a GPU count, priority class, and checkpoint settings.
Quotas and fairness: Each team gets guaranteed quota, with borrowing of idle capacity and fair-share ordering.
Topology-aware placement: Each gang lands in the best available block of the network.
Preemption and recovery: Urgent jobs can reclaim capacity, and evicted or failed jobs resume from checkpoints.
Reporting: Operators see utilization, wait time, and fairness metrics per team.
Nonfunctional requirements
Utilization: Keep GPU utilization above 80 percent without starving large jobs.
Scheduling latency: Decide each placement in under 100 ms.
Scalability: Manage 8,000 GPUs and about 1,900 jobs a day, with a queue of hundreds.
Availability: Replicated scheduler state with failover, and a scheduler outage must not kill running jobs.
Fairness: Delivered GPU-hours track quota and fair-share targets within a tolerance.
Recoverability: A hardware fault loses at most one checkpoint interval of work.
Resource estimation
Start with rough numbers before designing components. Assume a fleet of 1,000 nodes, each with 8 GPUs, for 8,000 GPUs total. That scale is large enough that naive scans and brute-force placement become expensive, yet small enough that centralized control with replicated state is still realistic.
Split jobs into broad bands so the scheduler can reason in classes rather than unique cases. Small jobs request 1 to 4 GPUs, medium jobs request 8 to 64 GPUs, and large jobs request 128 to 512 GPUs or more. Durations range from under an hour for experiments to multiple days for foundation-model training.
A typical ...