**The Scheduling Gap: Why AI Workloads Get Stuck in Cloud Clusters**
For years, cloud computing operated on a simple, quiet agreement between two control loops: the scheduler and the autoscaler. The scheduler’s job is to place workloads onto existing machines, while the autoscaler’s job is to add new machines when the current ones run out of space. Because compute was largely interchangeable, this division of labor worked perfectly. However, the rise of artificial intelligence and specialized hardware—like GPUs—has exposed a critical blind spot in this system.
Today, it is common to see a GPU workload stuck in a “pending” state, even though the cluster appears to have plenty of raw capacity. The reason? The autoscaler is answering the wrong question. It asks, “Should we add a node?” when the real question is, “Will the scheduler actually use that node?” When the autoscaler adds a machine that doesn’t match the specific needs of the pending workload, it leaves expensive accelerator capacity sitting idle.
Here is a look at why this mismatch happens, what the industry is doing to fix it, and how these two systems need to evolve together.
### The Right Shape vs. The Right Number
The core of the problem is a difference in perspective. An autoscaler typically looks at resource counts: how much CPU, memory, or how many GPU devices are available. But a modern accelerator is not just a simple chip; it is part of a complex ecosystem.
It relies on the CPU and local memory in its immediate vicinity, the network interface sharing its physical path, and high-speed links to its peer accelerators. This ecosystem is called topology, and it is a hierarchy. Devices connected on the same circuit board inside a server are fast; devices connected across a data center network are slower. Large AI models span many servers at once, so keeping these components close is critical.
Historically, Kubernetes described accelerators using simple counts—a node advertised a number, and a pod asked for some. This meant the scheduler and autoscaler were speaking different languages. The autoscaler might add a node with plenty of GPUs, but if those GPUs are isolated from the network paths the scheduler knows the workload needs, the pod remains stuck.
### Reservations and the Cost of Idle Hardware
Beyond topology, there is the issue of reserved capacity. Because accelerator hardware is scarce and expensive, teams often reserve it in advance. When a workload needs to scale up, it draws from this reserved pool.
The challenge is that the autoscaler often doesn’t know about these reservations. It sees an opportunity to add a node based on unreserved supply, not realizing the team has already paid for committed capacity on another node. This leads to wasteful spending, where the autoscaler provisions new hardware against unreserved capacity while ignoring the idle, reserved hardware that the scheduler is waiting to use.
### Sharing and Slicing the Pie
Modern accelerators are powerful enough that handing an entire device to a small job wastes most of its potential. To solve this, the industry has introduced two approaches: partitioning and consumable capacity.
Partitioning splits a single device into isolated, fixed instances so multiple jobs can run concurrently on the same hardware. Consumable capacity goes a step further, allowing several workloads to draw measured, fractional shares of a device simultaneously. The scheduler tracks these fine-grained allocations, but the autoscaler often struggles to weigh prospective nodes using the same fractional math, leading to mismatches in planning.
### The Need for a Shared Brain
Even if both loops understand the hardware using the same language, they still don’t act as one team. This is especially true for “gang scheduling,” an all-or-nothing scenario where a large job requires all its pieces to be scheduled simultaneously. If even one pod is missing, the entire job is held back, stranding hardware.
Currently, the autoscaler reasons pod by pod. It might add a node or two for a few pending members of a gang, wasting capacity because the job cannot start without the rest of its parts. The industry is moving toward “workload-aware scheduling,” which treats a group of pods as a single unit. However, the autoscaler has not yet caught up, creating a frustrating bottleneck where the system knows it has the capacity, but cannot trigger a full, coordinated scale-up.
### FAQ
**Q: Why do my GPU pods stay in a “Pending” state when the cluster looks full?**
A: A cluster can be out of capacity for a specific *type* of node, even if it has plenty of raw machines. If the autoscaler adds nodes that do not match the specific hardware requirements or topology the scheduler needs, the pods will remain pending. The scheduler and autoscaler are speaking different languages about what a usable node looks like.
**Q: What is the difference between partitioning and consumable capacity?**
A: Partitioning divides a single accelerator into fixed, isolated slices that run concurrently. Consumable capacity is more flexible, allowing multiple workloads to draw dynamic, measured fractions of a device at the same time.
**Q: What is the “all-or-nothing” problem in cluster scaling?**
A: This refers to gang scheduling, where a large AI job requires all its pods to be placed on nodes simultaneously. If the autoscaler only looks at pods individually, it might add a few nodes for a few pods, leaving the rest of the job stranded and unable to start, which wastes money and time.
**Q: How can teams avoid leaving expensive hardware idle?**
A: Teams should verify that their cluster’s scheduling and autoscaling systems share a common understanding of hardware topology and reservations. When a pod gets stuck, check if the autoscaler and scheduler agree on what a viable node looks like—rather than assuming the cluster has run out of capacity entirely.
### Conclusion
The mental model worth keeping is simple: one hardware model, two consumers. A scarce accelerator is only as useful as the agreement between the loop that schedules it and the loop that provisions it. Every place only one of them understands a capability is a place they can disagree and leave costly hardware idle.
As inference and agentic workloads become the norm, the patience for a “cold node” disappears. The industry is actively working to close the seam between scheduling and autoscaling, designing these systems together so that capacity is provisioned with the exact shape and topology the scheduler requires.
Thank you for reading
