The Evolution of AI Autoscaling on Kubernetes: How DRA and Kueue Are Transforming GPU Infrastructure Operations

\n

Software Infra: There Are GPUs, Yet the Job Won’t Start

If AI jobs remain Pending even when GPUs are available in the cluster, does that really mean there aren’t enough GPUs?

In AI infrastructure, the answer is often no. The problem isn’t the total number of GPUs, but the lack of GPU resources in a configuration that meets the job’s requirements.

Traditional Kubernetes autoscaling generally makes decisions like this:

Pod is Pending
→ Current nodes don’t have enough capacity
→ Add more GPU nodes

This model works well for general-purpose services focused on CPU and memory. But for large-scale training, distributed inference, and multi-GPU workloads, you can’t determine whether a job can be scheduled by asking only “How many GPUs are available?”

For example, a job requesting four GPUs may have much more complex requirements:

• Does it need four GPUs of the same model?
• Do the GPUs need high-speed interconnects such as NVLink?
• Do the GPUs and NICs need to be in a nearby NUMA region?
• Does it require a specific GPU partition or sharing mode?
• Do multiple Pods need to be ready simultaneously before the job can start?
• Must it run in a specific zone with reserved GPU capacity?

In other words, even if there are 10 GPUs left across the cluster, from the job’s perspective, there may be zero usable GPUs.

| Cluster state | Simple resource perspective | AI workload perspective | |---|---:|---| | 8 GPUs spread across multiple nodes | 8 GPUs available | May not be suitable for 8-GPU distributed training | | Enough GPUs, but NIC locality doesn’t match | Appears schedulable | Could result in poor communication performance or scheduling failure | | Only some GPUs remain as partitions | GPUs are available | Large jobs can’t start | | Only some GPU workers can be scheduled | Partial execution is possible | The entire gang workload can’t start |

Consider a distributed training job that requires all eight GPU workers to come up at the same time. If the autoscaler sees just one Pending Pod and adds a single GPU node, only some of the workers may start. But training won’t begin until the remaining workers are ready.

In this state, GPUs have already been allocated, yet no actual work is getting done. What’s left may be fragmented GPU capacity that’s difficult for other jobs to use, while costs continue to accrue.

8 GPU workers required
      ↓
Only 2 workers scheduled initially
      ↓
The remaining 6 are Pending
      ↓
Distributed training cannot start
      ↓
Allocated GPUs are occupied without generating useful output

This is a crucial distinction in operating Software Infra in the AI era. GPUs aren’t just interchangeable units of capacity. They are structured resources that must be considered alongside GPU models, interconnect topology, network paths, partitioning schemes, and reserved capacity.

So when you see a Pending status, the operations team shouldn’t immediately conclude, “We need more GPUs.” Instead, they should check:

• Do the requested GPU type and the GPUs actually available match?
• Are the required GPUs on the same node or within an appropriate high-speed fabric?
• Are the topology requirements for GPUs and network devices being met?
• Is this a gang workload that requires multiple Pods to be secured simultaneously?
• Are GPU partitioning or sharing policies leaving too little contiguous capacity?
• Do reserved capacity and the actual workload requirements align?

Ultimately, autoscaling for AI workloads isn’t about adding nodes. It’s about securing a hardware configuration that can actually run the job. If a job won’t start despite GPUs being available, what’s missing may not be GPU capacity, but the autoscaling capability to understand the topology and capacity conditions the workload requires.

Representing GPUs as Structure, Not Numbers: DRA and Software Infra

Can you properly schedule large-scale training with nothing more than a request for “4 GPUs”?

At first glance, that might seem like enough. But if you don’t know what GPU models they are, how fast they’re connected to one another, or how they’re linked to network interfaces, the number 4 GPUs leaves out critical information. As AI workloads grow more complex, Software Infra must understand not just how many resources are available, but how they’re structured and related.

The Limits of Number-Based GPU Requests

In conventional Kubernetes environments, GPUs are typically requested by specifying a quantity of extended resources.

resources:
  limits:
    nvidia.com/gpu: 4

This request communicates that “4 GPUs are needed.” But it doesn’t express the requirements of real-world distributed training and high-performance inference.

For example, it can’t answer questions like these:

• Do the GPUs need to be from a specific generation or model?
• Do all 4 GPUs need to be on the same server?
• Do the GPUs need to be connected by a high-speed interconnect such as NVLink?
• Do the GPUs and NICs need to be in the same NUMA domain?
• Can only a portion of each GPU be allocated?
• Does a specific GPU need to be allocated alongside a specific network device?

In other words, two nodes with 4 GPUs each can look identical under Kubernetes’ simple resource model. But one node may support high-speed communication between GPUs, while the other relies on slower PCIe or network paths. For large-scale model training, this difference can lead to more than degraded performance—it can cause job failures or long wait times.

What Is DRA?

DRA (Dynamic Resource Allocation) is Kubernetes’ approach to treating specialized hardware such as GPUs not as simple integer-valued resources, but as structured resources with attributes, relationships, and placement constraints.

The key is to move beyond the question “How many GPUs are there?” and ask:

Can we secure a set of devices that simultaneously satisfies this workload’s GPU, network, and topology requirements?

In a DRA environment, workloads can express the characteristics they need from accelerators in greater detail through ResourceClaims. The exact implementation and level of support vary by environment, but conceptually, DRA can handle requirements such as:

- A specific GPU model or accelerator class
- A set of GPUs connected to the same high-speed fabric
- Locality requirements between GPUs and NICs
- NUMA or PCIe topology constraints
- GPU partitioning or shared-capacity requirements
- Relationships that require specific devices to be placed together

The scheduler can then look for a combination of devices that meets the workload’s execution requirements, rather than simply finding a node with enough unallocated GPUs.

Why Topology Determines AI Performance

In distributed training, multiple GPUs constantly exchange gradients, parameters, and activation data. If the communication paths between GPUs are slow, overall training speed can drop significantly—even when the GPUs themselves deliver high compute performance.

Consider these two configurations:

| Configuration | GPU count | Connectivity | Expected outcome | |---|---:|---|---| | Node A | 8 | High-speed links between GPUs, good NIC locality | Well suited for large-scale training | | Node B | 8 | Slow communication paths between GPUs, far from NICs | Potential communication bottlenecks and degraded performance |

Under the conventional approach, both nodes are considered to have 8 GPUs. DRA, by contrast, is designed to let resources be evaluated based on GPU type, connectivity, and network relationships.

This distinction matters especially for the following workloads:

• Large-scale LLM pretraining and fine-tuning
• Multi-node distributed training
• Model parallelism requiring high-bandwidth communication
• Distributed inference where low latency is critical
• Data processing tasks that require proximity between GPUs and RDMA NICs

Why DRA-Aware Autoscaling Matters

The value of DRA doesn’t stop at the scheduler. Autoscalers need to understand the same hardware semantics, too.

Conventional autoscaling generally works like this:

Pod is Pending
→ Determine that GPU nodes are in short supply
→ Add GPU nodes

But a Pod may be Pending for reasons beyond a simple GPU shortage. The workload may not just need 4 GPUs—it may need 4 specific GPUs connected to the same fabric, with the right NIC locality.

A DRA-aware autoscaler accounts for this distinction:

Analyze the workload’s ResourceClaim
→ Check GPU model, topology, and sharing requirements
→ Select the appropriate node type
→ Provision suitable capacity
→ Scheduler allocates the actual set of devices

Problems can arise if an autoscaler doesn’t understand DRA information. It may create nodes that have enough GPUs but don’t meet the workload’s requirements. As a result, GPU costs go up while the Pod remains Pending.

Resource Semantics Matter for GPU Sharing, Too

DRA matters not only in environments where GPUs are exclusively allocated, but also for shared models such as GPU partitioning, time-slicing, and consumable capacity.

For example, imagine a single GPU being divided up like this:

Total GPU capacity: 100
Inference service A: 30
Batch job B: 40
Remaining capacity: 30

In this situation, Software Infra can’t simply ask, “Is the GPU in use?” It also needs to consider whether the remaining 30 units of capacity can actually be allocated to a given workload, whether performance isolation will be maintained, and whether enough contiguous capacity remains for a large job.

GPU sharing, then, isn’t merely a cost-saving feature. It’s an operational challenge that requires the scheduler, device plugin, autoscaler, and cloud provider to share the same understanding of resource semantics.

Questions to Ask from an Operational Perspective

If you’re considering adopting DRA or expanding an AI cluster, start by asking:

• Do your Kubernetes environment and autoscaler support DRA ResourceClaims?
• Can you collect information about GPU models, NVLink, PCIe, NUMA, and NIC locality?
• Does your autoscaler account for node topology—not just the number of GPUs?
• Do the scheduler and autoscaler interpret GPU partitioning and sharing models consistently?
• Is the reserved GPU capacity actually compatible with the workload’s topology requirements?

DRA is more than an upgrade to GPU request syntax. It signals a fundamental shift in AI-era Software Infra: moving beyond the abstraction of hardware as a number, toward a structured resource model that accounts for both performance and placement.

Software Infra: Scale the Entire Job, Not Just One Pod

Consider a distributed training job that cannot start until all eight GPU workers are ready. If a traditional autoscaler sees Pending Pods and adds nodes one at a time, some workers may start running while the rest remain stuck waiting.

The problem is that even if some workers are running, the training job as a whole cannot start. The allocated GPUs sit idle without doing any real computation, and other jobs can’t easily use those resources. It’s a classic form of AI infrastructure waste: nodes are added, but throughput doesn’t increase—only costs do.

The Limits of Pod-Level Scaling

A typical Kubernetes autoscaler uses the Pending state of individual Pods as its scaling signal.

One Pod is Pending
  → Add one node
  → Schedule some Pods
  → Other Pods remain Pending

But this approach doesn’t work for jobs that require multiple components to start together, such as distributed training, large-scale batch inference, and parameter server architectures.

For example, suppose a job requires:

8 GPU workers
2 parameter servers
High-speed network connectivity
All components to start together

What happens if only three workers are scheduled first?

• The job as a whole cannot start.
• The three allocated GPUs sit idle.
• GPUs and network topology for the remaining workers may not be available.
• Other workloads can’t use the capacity that’s already tied up.
• The autoscaler may keep scaling up in response to individual Pending Pods.

In other words, the job is stuck not because there aren’t enough GPUs, but because the capacity needed to accommodate the entire job hasn’t been secured all at once.

Gang Scheduling: Everyone Is Ready, or No One Starts

The key concept for solving this problem is Gang Scheduling. Instead of treating multiple Pods as independent scheduling units, it groups them into a single logical workload.

The principle behind Gang Scheduling is simple:

Start the job only when all required Pods can be scheduled.

This prevents some Pods from occupying GPUs while the rest of the job is unable to run.

Traditional approach
3 workers running + 5 workers waiting
→ Job does not start
→ 3 GPUs sit idle

Gang Scheduling
Check whether all 8 workers and supporting components can be accommodated
→ If yes: Admit and schedule the entire job
→ If no: Keep the entire job waiting

At first glance, “waiting” may seem inefficient. In practice, however, it prevents Pods that can’t run the job from holding onto GPUs, making the cluster much more efficient overall.

Why Workloads and PodGroups Matter

To tell Kubernetes to treat these components as a single unit, we need a way to represent the Pods as a group. This is where Workloads and PodGroups come in.

A PodGroup can contain information such as:

• The number of Pods that must start together
• The minimum number of replicas required to run
• The required GPU, CPU, and memory capacity
• Priority and queue information
• Specific topology or network requirements
• Admission conditions for the entire job

The scheduler can then ask not, “Where should I place this Pod?” but:

Can this entire job be accommodated at once, with
8 GPUs,
the required network connections,
the same zone or fabric conditions,
and all required supporting Pods?

This perspective is essential to Software Infra operations. An AI workload’s true unit of execution is not one Pod, but the entire job—a combination of multiple Pods, GPUs, and network resources.

Autoscalers Also Need to Understand the Entire Job

However, the problem isn’t automatically solved just because the scheduler understands Workloads or PodGroups. If the autoscaler still looks only at the Pending signal from individual Pods, it may miscalculate the capacity required.

Scheduler:
"All 8 GPU workers are required."

Autoscaler:
"I see one Pending Pod, so I’ll add one GPU node."

To bridge this gap, we need provisioning requests. When a Workload explicitly communicates its total capacity requirements, the autoscaler and cloud provider can prepare infrastructure for the job as a whole, rather than for individual Pods.

Define the Workload
  → Calculate total GPU, CPU, and network requirements
  → Create a provisioning request
  → Secure the required node capacity
  → Confirm that capacity is ready
  → Admit the entire workload
  → Schedule all components together with gang scheduling

This flow is far more predictable and cost-effective than reactive scaling that adds nodes only after Pods get stuck.

The Role of Kueue

When multiple teams and AI jobs share a GPU cluster, simply adding capacity isn’t enough. The cluster must also manage which jobs run first, whether they stay within quota, and whether large jobs can secure the resources they need.

This is where Kueue handles workload admission and quota management.

• Manage GPU quotas by team or project
• Control job priority through queues
• Admit only workloads for which full capacity is ready
• Fairly manage the wait for large gang workloads
• Prevent incomplete resource allocations

Together, Kueue and provisioning requests turn the cluster from a simple Pod execution environment into something closer to a control system that coordinates GPU capacity at the job level.

Operational Benefits

Workload-aware autoscaling can help address issues such as:

| Operational problem | How job-level autoscaling responds | |---|---| | Some workers run while their GPUs sit idle | Admit the job only when the entire workload can run | | Nodes are added piecemeal as individual Pods scale up | Calculate capacity based on the entire workload | | Large training jobs remain stuck in the queue for a long time | Manage quotas, priorities, and capacity readiness together | | GPU costs rise, but training never starts | Allocate resources only after confirming the job can run | | Multiple teams compete heavily for GPUs | Apply queue and quota policies through Kueue |

Ultimately, autoscaling in AI environments isn’t about counting Pods. It’s about determining whether the job can run right now, with all of its requirements met. When Software Infra supports admission and provisioning at the workload level, GPUs become more than partially allocated costs: they become resources that deliver real performance and throughput.

Kueue and Provisioning Requests from a Software Infra Perspective: Securing the Capacity You Need Up Front

If you can specify in advance how many GPUs a workload needs and what constraints it must meet, why wait until Pods are Pending one by one before adding nodes?

Traditional Kubernetes autoscaling is generally reactive: it responds after scheduling fails. The Scheduler fails to place a Pod, and once Pending Pods pile up, the Autoscaler detects them and adds nodes. This is practical enough for CPU- and memory-centric services, but it can be too slow and imprecise for AI workloads that require multiple GPUs and high-speed networking at the same time.

Kueue and Provisioning Requests aim to change this sequence. Instead of reacting to the results of failed scheduling, they let a workload declare and secure its full capacity requirements before it runs.

The Limits of Pod-Level Reactive Autoscaling

Suppose a distributed training job requires the following:

• 8 GPU workers
• At least 1 GPU per worker
• The same zone or high-speed network fabric
• 2 parameter servers
• All components to start at the same time

With the traditional approach, some worker Pods may be scheduled first while the rest remain Pending. The Autoscaler adds nodes in response to Pending Pods, but there is no guarantee that the new nodes will meet the required GPU model, topology, and network constraints.

This can lead to inefficiencies such as:

• The job cannot start even though some GPUs have already been allocated
• Partially scheduled Pods interfere with the placement of other workloads
• GPU nodes are added sequentially, in smaller numbers than needed
• More GPUs are added, but not enough to accommodate the entire gang workload
• Expensive on-demand nodes are created instead of using reserved capacity

In other words, the problem is not simply a shortage of GPUs. It is that the workload’s requirements as a whole are not communicated during capacity provisioning.

Kueue: A Layer for Managing Admission Before Execution

Kueue is a component for managing quota, queues, and admission for batch and AI workloads in Kubernetes. Its core role is to decide whether a job can run now at the workload level, rather than Pod by Pod.

Kueue can make admission decisions based on:

• GPU quotas by organization or team
• Priority and queue policies
• Required CPU, memory, and GPU resources
• Gang-scheduling requirements, where multiple Pods must be ready at the same time
• Reserved capacity and available cluster resources
• Specific accelerator or topology requirements

The important point is that Kueue is more than a simple queueing tool. By delaying admission until an AI job can secure the resources it needs to run, it helps reduce situations where only some Pods start, leaving the workload in an incomplete state.

Traditional flow

Create Pods
  → Scheduler attempts placement
  → Some Pods remain Pending
  → Autoscaler adds nodes
  → Scheduler tries again


Kueue-based flow

Submit workload
  → Check quota and priority
  → Request full capacity
  → Prepare required nodes
  → Admit the entire workload
  → Scheduler places the Pods

This structure is especially important from a Software Infra operations perspective. GPUs are expensive and take time to provision, so connecting capacity provisioning with admission is far more predictable than placing Pods first when it is uncertain whether they can actually run.

Provisioning Requests: The Link for Communicating Full Requirements to the Autoscaler

A Provisioning Request is a mechanism for explicitly communicating a workload’s capacity needs to an Autoscaler or cloud provider.

The key question is not “How many Pods are Pending?” but “What kind of infrastructure, and how much of it, does this job need to run?”

For example, a request may specify requirements such as:

Need nodes with 8 GPUs, or equivalent capacity
Must be placed in the same zone
Require a specific GPU model
Require a high-speed fabric between GPUs
Require locality with high-performance NICs
Prioritize reserved GPU capacity
Hold admission until the entire workload is ready

When this information is passed to the Autoscaler, it can provision nodes that meet the workload’s execution requirements, instead of simply looking for the cheapest or quickest nodes to create.

The overall flow can be understood as follows:

Define workload requirements
   ↓
Kueue evaluates quota, priority, and admission conditions
   ↓
Create a Provisioning Request
   ↓
Autoscaler or Cloud Provider secures the required capacity
   ↓
Return ready status
   ↓
Kueue admits the entire workload
   ↓
Scheduler places Pods as a PodGroup or gang

This approach improves on the traditional structure, where the Scheduler and Autoscaler had to infer each other’s decisions. The Scheduler can wait for an environment in which placement is possible, while the Autoscaler prepares capacity based on the workload’s actual requirements.

“Securing Capacity Up Front” Does Not Always Mean Atomic Provisioning

However, having a Provisioning Request does not guarantee that all GPUs and nodes will be secured atomically—in other words, perfectly all at once.

Real-world cloud environments involve variables such as:

• Shortages of specific GPU models in particular regions
• Topology mismatches in reserved capacity
• Differences in node creation times
• Differences in Provisioning Request support across providers
• Differences in DRA and topology awareness across Autoscaler implementations
• Delays in the actual supply of infrastructure even after quota checks pass

So, rather than a single, fully integrated control loop, today’s approach is closer to a best-effort integration model that connects previously separate control loops more tightly.

Even so, the shift is clear. Where the traditional approach scales up capacity after observing failed Pods, Kueue and Provisioning Requests are moving Kubernetes toward a model that plans the workload first, prepares the required capacity, and then starts the workload.

Policies for Operations Teams to Define

When adopting Kueue and Provisioning Requests, simply enabling the technical features is not enough. Teams also need to define policies in the following areas:

| Policy area | What to consider | |---|---| | Quota | How should GPU usage limits be divided by team, project, and environment? | | Priority | Which should take precedence: urgent inference jobs or long-running training jobs? | | Admission | Can a job wait until all required capacity is secured? | | Reservation | Which workloads should get priority for reserved GPUs? | | Topology | To what extent should same-node, same-rack, or same-zone placement be enforced? | | Timeout | Should a job be canceled or retried if capacity provisioning is delayed for a long time? | | Observability | Can you tell whether a job is waiting because of quota, topology, or a supply shortage? |

In particular, operational dashboards should distinguish between the following states, rather than simply showing the number of Pending Pods:

• Waiting for admission because of insufficient quota
• GPU capacity being provisioned
• Unable to find nodes that meet topology requirements
• Reserved capacity that does not match the workload’s requirements
• Waiting for capacity for the entire gang to become available

Without these distinctions, operations teams may assume they are short on GPUs. But the actual cause may not be the total number of GPUs. It could instead be the admission policy, incorrect topology requirements, or a mismatch in how reserved capacity is distributed.

Kueue and Provisioning Requests take Kubernetes a step beyond being a simple Pod execution platform. For AI Software Infra, planning and securing the resources a workload needs before it runs—rather than adding nodes after Pods fail—is becoming a far more important operating model for cost, reliability, and GPU utilization.

From GPU Counts to AI Infrastructure Optimization: Cost and Performance Decisions in Software Infrastructure

Is adding more GPU nodes always the right answer? If expensive on-demand instances are purchased while already-reserved GPUs sit idle—or if new nodes don’t meet the communication requirements of distributed training—autoscaling can drive up costs and latency instead of solving performance problems.

Autoscaling in AI environments can no longer be reduced to GPU shortage → add nodes. What matters is not the number of GPUs, but whether you have the right capacity to run the workload.

For example, a distributed training job that requires eight GPUs won’t run just because eight GPUs exist. It also needs high-speed interconnects between GPUs, good locality between GPUs and NICs, placement in the same zone, and sufficient network bandwidth. Adding multiple GPU nodes with slow communication paths may get the job started, but can severely degrade performance during distributed communication steps such as all-reduce.

From a cost perspective, reservation-aware strategies are also essential. Operations teams should evaluate capacity in this order:

1. Check whether any purchased or reserved GPU capacity is still available.
2. Verify that the reserved capacity meets the workload’s GPU model, topology, and zone requirements.
3. If reserved resources can’t meet those requirements, provision the minimum amount of on-demand capacity in the required configuration.
4. When the job finishes, apply scale-in policies so unnecessary GPU costs don’t continue to accrue.

Common scenarios that call for this kind of judgment include:

| Situation | Result of simple autoscaling | Optimized approach | |---|---|---| | Reserved GPUs are available, but in a different zone | Immediately add on-demand GPUs | Compare latency requirements and transfer costs to determine whether to use the reserved resources | | There are enough GPUs, but NVLink or NIC requirements aren’t met | Add more GPU nodes | Provision only node types that meet the topology requirements | | A small inference job occupies an entire GPU | Low GPU utilization and unnecessary scale-out | Use partitioning or shared capacity | | Only some of the Pods for a distributed training job can be scheduled | The job is delayed while only some GPUs are allocated | Secure capacity for the workload as a whole, then admit it all at once |

This isn’t the responsibility of a Kubernetes autoscaler alone. The scheduler, DRA-based resource claims, Kueue’s quota and admission policies, and the cloud provider’s reserved-capacity information all need to work together. In other words, from a Software Infrastructure perspective, AI autoscaling is less about adjusting node counts and more about a control system that jointly optimizes hardware topology, workload requirements, and cost policies.

As a result, the metrics operations teams track need to change, too. GPU utilization alone isn’t enough. They should also monitor reserved GPU utilization, the percentage of workloads Pending due to topology mismatches, provisioning time by workload, gang scheduling wait time, and the cost savings achieved after scale-in.

The goal of good autoscaling isn’t to add as many GPUs as possible. It’s to secure GPU capacity with the right interconnects and cost profile when it’s needed—and reclaim it quickly when the job is done.