Ai2's new GPU scheduler delivers 98% of owed hours

Ai2 runs thousands of NVIDIA H100, B200, and B300 GPUs across clusters ranging from 88 to 1024 GPUs, serving about 150 internal researchers. Demand for GPU time has consistently exceeded supply, with outstanding requests for 2-3x more GPUs than are available. The old priority-based scheduler let workloads opt out of preemptability, which led to GPU 'squatting' and priority inflation, where eventually 100% of scheduled workloads used HIGH priority. On-call engineers spent most of their ticket time negotiating shutdowns of non-preemptable workloads on hosts needing maintenance.

The new budget-based system assigns each project a guaranteed share of GPU time, such as Project A1 having a 35% claim on total capacity. A hierarchical fair-share scheduler tracks occupancy over a 7-day lookback window and sorts workloads accordingly. Workloads must declare a minimum runtime in exchange for preemption protection; minimum runtime can be set to zero for free but preemptible use. The rollout began at the end of July, cluster by cluster.

Simulations using historical data and constructed scenarios predicted debug workload wait times would fall from about 6 hours to 5 minutes, and real outcomes overperformed, with p90 waits dropping from 2 hours to 30 seconds. Occupancy held steady at 98% before and after, with 18% of delivered GPU time unallocated. The learning curve was steeper than assumed, with live explanatory sessions proving more effective than documentation alone.