Skip to main content

How LLM-D 1.0 Is Transforming Kubernetes LLM Inference Infrastructure: Key Technologies and Performance

Created by AI\n

Software Infrastructure: The GPUs Are Busy, So Why Are Responses Still Slow?

Users may think they’re asking the same question, yet some requests get answered almost instantly while others take much longer. GPU utilization is high, and you’ve added plenty of servers. The cause may not simply be a lack of GPU capacity.

The key issue is that requests are scattered across servers that have no memory of the previous conversation.

Before generating a response, an LLM must first read the user’s prompt and conversation history. The intermediate state created during this process is called the KV cache (Key-Value Cache). If the next request in the same conversation is routed back to the server that handled the previous one, it can reuse the KV cache already stored there. But if the request goes to a different server, the previous context has to be recomputed.

Typical load balancing

User request A → Replica 1
User request B → Replica 2
User request C → Replica 3

This approach is effective at distributing load evenly across servers. But it can cause problems in environments where conversations continue over multiple turns, as with agents, or where long documents are referenced repeatedly. Even if Replica 1 has already computed the conversation context, the GPU has to process the same prompt all over again if the next request goes to Replica 2.

The results are clear:

  • The time it takes to generate the first token—TTFT (Time to First Token)—increases.
  • GPU compute and memory are used redundantly.
  • The longer the context, the greater the waste.
  • Some servers may sit idle while holding cached data, while others become overloaded by recomputing it.
  • Users experience inconsistent service quality.

This is a problem that existing Kubernetes Services and conventional round-robin load balancers struggle to solve. They generally distribute requests based on questions like, “Which servers are healthy?” and “Which servers have the lightest load?” But LLM inference calls for one more question:

Which server already has the context for this request?

Released in October 2026, llm-d 1.0 puts that question front and center. Rather than replacing model execution engines like vLLM or SGLang, llm-d acts as a software infrastructure control plane that brings multiple model servers and GPU/TPU pools together into a single intelligent inference system.

Instead of simply looking for an idle server, the llm-d router considers:

  • Where KV caches containing the same prompt prefix are located
  • Which servers previously handled the same agent or user session
  • The current load on each replica
  • Expected response latency
  • Tenant priorities and service policies
  • The configuration and status of accelerator pools, including GPUs and TPUs
Cache-aware routing with llm-d

Agent Turn 1 → Replica A
Agent Turn 2 → Replica A  ← Reuse existing KV cache
Agent Turn 3 → Replica A  ← Preserve conversation context

But if Replica A is saturated:
Agent Turn 3 → Replica B  ← Reroute to prevent response failures

The important point is that llm-d doesn’t insist on sticky sessions at all costs. Reusing the cache is faster, but if a particular replica is overloaded, it can actually make overall responsiveness worse. llm-d therefore tries to find the best balance between cache locality and cluster-wide load balancing.

This shift changes how we think about LLM infrastructure. In the past, the focus was on “how to run a faster model server.” Now the key questions are “how to reuse context already computed across the cluster” and “which accelerator should handle each request.”

If responses are slow even while the GPUs are busy, there’s something to check before adding more GPUs: Are requests going to the server that remembers the most context, rather than simply the server that’s fastest?

Software Infra: It Starts with Where You Send the Request—Routing That Understands KV Cache

Is sending a request to the least busy server always the fastest option?

Not necessarily for LLM inference. In agent environments where conversations span multiple turns, a server that has already processed the previous conversation context can be far more advantageous—even if it’s a little busier. That’s because its GPU memory still holds the KV cache generated while processing the earlier prompt.

In a typical Kubernetes environment, conventional load balancing usually works like this:

Request 1 → Replica A
Request 2 → Replica B
Request 3 → Replica C

This approach is effective at distributing load evenly across servers. But it creates a problem when Request 2 is a follow-up in the same agent session as Request 1. Replica A already has the KV cache for the context it needs, but Replica B doesn’t know that. As a result, B has to read through the long conversation history and recompute the prompt.

The consequences are clear:

  • Increased TTFT (Time to First Token)—the time it takes for the first token to appear
  • Increased GPU compute and memory usage
  • Lower throughput in environments with long contexts
  • Delays to the agent’s next task, slowing down the overall workflow

This is where llm-d’s core idea begins. Rather than simply finding “the least busy replica right now,” the platform factors which replicas hold which prefixes and session contexts into its routing decisions.

Agent Turn 1 → Replica A
Agent Turn 2 → Replica A  ← Reuse existing KV cache
Agent Turn 3 → Replica A  ← Preserve continuity across the conversation

This approach is commonly called prefix-cache-aware routing or session-aware routing. The router checks the request’s prefix, session information, and cache location, then prioritizes the endpoint that can make the best use of the existing cache.

But the presence of a cache doesn’t mean a request should always go to the same server. If Replica A is heavily saturated, the cost of waiting in its queue may outweigh the benefit of reusing the cache. That’s why llm-d’s routing considers several factors together:

| Factor | What it means for routing | |---|---| | Prefix cache location | Can the results of earlier prompt computation be reused? | | Session affinity | Is this the server that handled the same agent’s previous turn? | | Current load | What are the request queue, GPU utilization, and workload in progress like? | | Expected latency | Which option is faster when both cache-reuse gains and wait time are taken into account? | | Tenant priority | What priority does the service policy assign—premium, standard, batch, and so on? | | Accelerator topology | Is the request a good fit for the type of GPU or TPU and the characteristics of the pool? |

In other words, llm-d doesn’t mechanically choose between “the server with the cache” and “the least busy server.” The key is to weigh the compute savings from cache locality against the wait time caused by current load, and find the better option.

The process can be simplified as follows:

New request arrives
   ↓
Find replicas holding a KV cache for the same prefix or session
   ↓
Check candidate replicas’ queues, load, and expected latency
   ↓
Compare the benefits of cache reuse against the cost of waiting
   ↓
Route the request to the most favorable endpoint

This approach is especially effective for services that share long system prompts, coding agents that use an entire codebase as context, and workflow automation agents that run for dozens of turns or more. Instead of prefilling tens of thousands of tokens of conversation history with every request, they can continue from already-computed state wherever it’s available.

Of course, session affinity is not an absolute rule. If a replica is unhealthy, its queue is getting too long, or it needs to handle higher-priority requests, llm-d may relax affinity and move the request to another replica. In these cases, preventing request failures and maintaining responsiveness take priority over a cache hit.

This balancing act is what sets llm-d apart from a traditional load balancer. Traditional Software Infra focuses on server availability and traffic distribution; llm-d also seeks to understand the value of the context inside an LLM request and where its computed state resides.

Ultimately, the key question in an inference system is no longer just, “Which server is free?”

“Which server already knows this request’s context—and does it still have capacity to handle it?”

llm-d’s KV-cache-aware routing is a distributed inference control layer designed to answer exactly that question.

Prefill and Decode Disaggregation from a Software Infra Perspective

Reading a long question and writing an answer one token at a time are not the same kind of task. Yet until now, many LLM services have handled both within a single replica and a single GPU resource strategy.

This approach is simple, but its bottlenecks become apparent when requests from agents with long contexts and real-time conversations arrive at the same time. To address this, llm-d’s disaggregated serving separates inference into Prefill and Decode, with a dedicated pool for each stage.

Traditional approach

One Replica
 ├─ Prompt processing (Prefill)
 └─ Token generation (Decode)
      ↓
  Both share the same GPU resources


llm-d disaggregated approach

Prefill Pool ─────▶ Decode Pool
  Compute-intensive       Memory- and latency-intensive

Prefill: Reading Long Inputs and Building the KV Cache

Prefill is the process of feeding the user’s entire prompt into the model. For example, if the model needs to process a 50,000-token document, the context of a code repository, or a history of previous conversations, it must analyze all the input tokens and perform attention computations.

This stage has several defining characteristics:

  • The longer the prompt, the more rapidly the compute requirements grow.
  • High GPU throughput is important.
  • Large-batch processing and parallelization efficiency have a major impact.
  • The process produces the KV Cache needed for subsequent generation.

The problem arises when a prefill task for a long prompt comes in. If the same replica is already generating a real-time response, the heavy prefill workload can consume GPU resources and slow down the ongoing conversation as well. Users may feel as if the response has stalled, while service operators see P95 and P99 latency worsen.

Decode: Generating a Response One Token at a Time

Decode is the iterative process that follows prefill, in which the model predicts and generates the next token. From a short response like “Hello” to hundreds of lines of generated code, every output token is produced at this stage.

Decode has different resource characteristics from prefill:

  • Because tokens are generated sequentially, low latency is important.
  • It repeatedly reads the KV Cache generated earlier.
  • HBM capacity and memory bandwidth matter more than raw compute performance.
  • User experience depends heavily on smooth generation after the first token.

In other words, if prefill is like “quickly reading a large volume of documents,” decode is more like “writing one sentence at a time while continually referring to notes.” When both are tied to the same resources, their performance characteristics inevitably conflict.

How llm-d Separates the Two Stages

Through DisaggregatedSet, llm-d manages the prefill and decode pools as a single logical deployment unit. Operators can configure the two pools independently and apply different scaling strategies suited to each workload.

Client Request
      │
      ▼
llm-d Router
      │
      ├─ Process long prompt → Prefill Pool
      │                           │
      │                           └─ Transfer KV Cache
      │
      └────────────────────────▶ Decode Pool
                                    │
                                    ▼
                              Streaming Response

In this architecture, the router does more than simply forward requests. It selects the appropriate endpoint based on factors such as the request session, prefix cache location, current pool load, and expected latency. The KV Cache produced during prefill also needs to be transferred and placed so it can be used during decode.

That makes KV cache tiering across GPU HBM, host RAM, SSDs, and external storage an important part of the Software Infra layer. This is not simply a matter of splitting servers in two—it is a matter of separating computation from state while preserving context locality.

Operational Benefits of Independent Scaling

Separating prefill and decode makes it possible to scale each stage differently according to traffic patterns.

| Scenario | Traditional single-replica architecture | llm-d disaggregated architecture | |---|---|---| | Spike in requests with long documents or long contexts | All replicas may slow down | Scale out the prefill pool | | Spike in short queries and long responses | GPU resource contention | Scale out the decode pool | | Real-time APIs and batch workloads coexist | Real-time requests may be delayed | Separate pools and priorities through QoS policies | | Mixed GPU types | Deployment configuration becomes complex | Apply accelerator strategies by stage | | Model updates | Compatibility must be managed across components | Manage transitions through coordinated rollouts |

For example, in an internal coding agent that repeatedly reads a large codebase, prefill workloads can become very heavy. By contrast, for a customer support chatbot that generates long answers from short inputs, decode latency may be the more important metric.

llm-d does not collapse these differences into a single, average GPU utilization figure. Instead, it points toward assigning compute-throughput-oriented resources to prefill and memory-bandwidth and response-stability-oriented resources to decode.

Disaggregation Is Not Always the Right Answer

That said, disaggregated serving is not necessary for every service. Because the KV Cache must be moved or shared between the prefill and decode pools, it adds complexity to the network, storage, and cache-transfer layers. Operators must also account for per-pool autoscaling, failure handling, and version compatibility.

This approach is especially promising in environments with:

  • Agent services that use long contexts
  • Large-scale multi-turn sessions
  • A mix of interactive traffic and batch inference
  • Fleets of dozens to hundreds of GPUs or TPUs
  • Significant imbalances between prefill and decode workloads
  • APIs with strict P95 and P99 latency requirements

By contrast, for a service with a single model, a small number of replicas, and short, predictable requests, a standard vLLM deployment may be simpler and more efficient.

Ultimately, llm-d’s prefill/decode disaggregation is not about “using more GPUs.” It is a Software Infra strategy for achieving more predictable latency and higher resource efficiency across the same accelerator fleet by separating inference workloads with different characteristics.

The Era of Shared GPUs: Scheduling for Priority and Fairness | Software Infra

When premium APIs, coding agents, batch inference, and RL rollouts all share a GPU fleet, which requests should be handled first?

In a simple FIFO (First In, First Out) queue, requests are served in the order they arrive. The problem arises when one tenant suddenly sends a large burst of requests. Low-priority batch jobs can occupy the GPU queue, delaying real-time user requests that arrive afterward. Before long, one team’s burst becomes a noisy neighbor problem, driving up TTFT and P95 latency across the entire service.

llm-d 1.0 treats this not merely as a matter of allocating GPU resources, but as a Software Infra problem: applying policies to schedule inference requests themselves.

Not All Requests Matter Equally—even on the Same GPU

A single accelerator cluster can host workloads with very different characteristics.

| Traffic type | Typical requirements | Example priority | |---|---|---| | Premium API | Low TTFT, strict SLOs | High | | General user requests | Balanced response times | Normal | | Coding agents | Multiple turns, potential bursts | Medium to high | | Batch inference | High throughput; latency is acceptable | Low | | RL rollouts | Make use of idle resources; run at scale | Low |

FIFO alone cannot express these differences. For example, just because an RL rollout has queued thousands of long requests first doesn’t mean a real-time API request from a paying customer should be pushed to the back of the line.

Instead of assigning requests a simple sequence number, llm-d assigns flow keys that let it apply policies:

  • Fairness ID: Identifies which tenant or user group the request belongs to
  • Priority Band: Distinguishes priority levels based on service tier
  • Workload or Program ID: Identifies the application or workflow associated with the request

This information helps the router decide not only “which replica should receive the next request?” but also “should this request be sent to the backend right now?”

Raising Priority Alone Isn’t Fair

Focusing only on high-priority traffic creates another problem. If premium traffic keeps pouring in, standard tenants may effectively lose their chance to be served. On the other hand, treating every request exactly the same makes it difficult to guarantee SLOs for critical real-time services.

llm-d’s flow control addresses both goals:

  1. Handle higher-priority requests first.
  2. Apply fairness across tenants within the same priority level.
  3. Control request flow at the router before backend queues become excessive.
  4. When capacity is limited, delay lower-priority workloads or limit some requests.

In other words, the system handles “the most important requests first” while ensuring that, within each tier, no single tenant can monopolize the GPUs. This may sound similar to setting CPU and memory quotas in Kubernetes, but it’s more complex in practice: LLM inference also requires accounting for token generation rates, KV cache usage, and expected latency.

Under Overload, Don’t Make Everyone Wait

One of the most dangerous situations in LLM inference occurs when requests arrive faster than they can be processed. The longer the queue grows, the longer users wait. Once timeouts kick in, they begin retrying—and those retries add even more load, amplifying the incident.

In these situations, llm-d can use probabilistic admission control. Rather than sending every request to the backend unconditionally, the system can gradually delay or limit some lower-priority requests based on how saturated it is.

Low load
Request → Router → Backend queue → GPU

High load
Premium requests → Prioritized
Standard requests → Queued according to fairness policy
Batch/RL requests → Delayed, limited, or shed

The goal is to prevent the entire system from collapsing, even if that means sacrificing some requests. For organizations where real-time APIs and batch jobs share the same GPU pool, this is more than a performance optimization—it’s an operational policy for service continuity.

Policies Aren’t Defaults—Operators Must Design Them

llm-d’s advanced QoS doesn’t automatically solve every problem. To remain compatible with existing environments, its default behavior is relatively close to first-come, first-served. Operators need to define the actual priority, fairness, and quota policies.

Before adoption, you should be able to answer at least these questions:

  • What is the P95 TTFT target for premium customers?
  • How long can batch inference be delayed?
  • What are the maximum concurrent requests and token budget for each tenant?
  • How much burst traffic should agent workloads be allowed to generate?
  • When GPUs are saturated, which requests should be delayed or rejected first?

Ultimately, llm-d doesn’t offer one “fair” answer. It provides a control layer for translating an organization’s business priorities into inference infrastructure policies.

In the era of shared GPUs, high utilization alone doesn’t make a system good. Critical requests must be handled on time; a sudden traffic spike from one tenant must not bring other services to a halt; and spare capacity should be used efficiently by batch and RL workloads. llm-d’s multi-tenant QoS is a core capability for managing this balance at the Software Infra level.

The Potential of an Inference Operating System from a Software Infra Perspective—and What to Check Before Adopting One

Should you decide to adopt a system based solely on an announcement that benchmark throughput improved several-fold? The answer calls for caution. llm-d’s core value is not simply making a single model server faster. It brings together GPUs, TPUs, storage, routers, queues, and caches so you can operate the entire cluster as a single inference system.

In other words, llm-d is less a standalone serving tool than a Software Infra operating layer for LLM workloads. But greater control also means greater operational responsibility.

What Changes for Software Infra: From Running Servers to Running an Inference System

Traditionally, teams have scaled out vLLM or SGLang replicas and distributed traffic with Kubernetes Services. This setup is simple and quick to get started with, but its limitations become clear as agent workloads and long-context requests grow.

llm-d aims to answer the following questions at the infrastructure level:

  • Which replica should handle this request so it can reuse the KV cache?
  • Is prefill work for long prompts interfering with real-time decode responses?
  • How can we ensure Premium tenant requests don’t get pushed behind batch inference?
  • When GPU HBM runs short, how far can we move the KV cache?
  • Can we keep routing the same agent session to the same execution location?

From this perspective, llm-d is a kind of inference operating system built on Kubernetes. Instead of looking only at CPU utilization or pod counts, it uses prefix-cache hit rate, queue saturation, TTFT, session locality, and tenant priority as operational signals.

This can be especially valuable when multiple teams share a single GPU fleet. A simple round-robin approach makes it difficult to get the most out of expensive accelerators.

Key Metrics for Evaluating the Impact of Software Infra

It’s not enough to evaluate llm-d’s performance with a single figure like “tokens/sec increased.” Adoption decisions should account for user experience, cost, and reliability together.

| Validation area | Questions to ask | |---|---| | TTFT | Has the time to generate the first token actually decreased? | | P95/P99 latency | Has latency improved under congestion, not just on average? | | Prefix-cache hit rate | Do session affinity and prefix-aware routing actually lead to cache reuse? | | Inter-token latency | Is the delay between generated tokens stable? | | GPU utilization | Has GPU utilization increased without sacrificing latency? | | Cost per GPU-hour | Does higher throughput translate into actual cost savings? | | Queue delay | How long do requests wait in the router and backend queues? | | Failover time | When a router or worker fails, are request loss and recovery time within acceptable limits? | | Cache eviction frequency | Have cache eviction storms caused by HBM shortages become less frequent? |

Performance figures from official announcements are useful as a reference, but results can vary significantly depending on model size, prompt length, GPU type, storage speed, and traffic patterns. For example, cache-aware routing can make a big difference for long, multi-turn agent workloads, while its benefits may be limited if most requests are short and independent.

Software Infra Complexity: More Responsibility Alongside More Control

With llm-d, the scope of operations expands from model servers to a distributed inference platform. Compared with a typical Kubernetes + vLLM setup, there are more components to manage:

  • High availability for the router and endpoint picker
  • Prefix-cache index and session affinity policies
  • Capacity planning for prefill and decode pools
  • KV cache tiering across HBM, CPU RAM, SSD, and external storage
  • Tenant-specific priority, fairness, and quota policies
  • Integration with autoscaling and queue systems such as KEDA, HPA, and Kueue
  • Accelerator topology and network bandwidth design
  • Observability based on Prometheus and OpenTelemetry
  • Compatibility between prefill and decode versions during rolling upgrades

Prefill/decode disaggregation is powerful, but not every organization needs it. Splitting prefill and decode into separate pools lets you scale each stage independently, but it also means accounting for KV cache transfer between pools, network latency, failure handling, and version consistency.

So llm-d is not a product that “automatically cuts costs once installed.” It is Software Infra that teams with mature operational capabilities can use to fine-tune performance and cost.

Often-Overlooked Software Infra Security and Governance Considerations

KV caches can retain sensitive context, including user prompts, conversation history, and system prompts. Security design becomes even more important when cache tiering moves data across CPU RAM, SSDs, and external storage.

At a minimum, check the following before adoption:

  • Where KV caches are stored and how they are encrypted
  • Cache isolation policies between tenants
  • Cache retention periods and guarantees of explicit deletion
  • Whether data is transferred across regions
  • Access controls and audit logs when using external storage
  • Data residency requirements for personal, financial, and medical information
  • How cache data is handled during failures, restarts, and scale-downs

Strategies to increase cache reuse for performance can conflict with the principle of data minimization. Platform, security, and data governance teams should therefore work together to define cache lifecycle policies.

Where Software Infra Is a Good Fit—and Where to Be Cautious

llm-d is worth prioritizing for evaluation in environments where:

  • Multiple tenants share a large GPU or TPU fleet
  • Multi-turn requests are common, as with coding, research, or customer-support agents
  • Long contexts and repeated system prompts are common
  • vLLM or SGLang is already in production
  • SLOs need to be separated for Premium, Standard, and Batch workloads
  • GPU costs and recomputation costs caused by cache misses are high
  • Prefill and decode need to be optimized on separate accelerator pools

On the other hand, if you’re running just one model with a small number of replicas, handling short requests, and seeing little session reuse, a standard Kubernetes Service and vLLM setup may be a better fit. In that case, llm-d’s control features could become an unnecessarily complex operational layer.

Recommended Validation Approach Before Adopting Software Infra

The most practical approach is to validate llm-d in stages against a limited workload, rather than switching everything over at once.

First, establish a baseline for your existing serving environment. Measure TTFT, P95 latency, GPU utilization, cache hit rate, and cost per GPU-hour, then compare them against llm-d using the same model and traffic.

Next, reproduce real operating conditions. In addition to average throughput under normal conditions, be sure to include scenarios such as:

  • A surge of long prompts arriving at the same time
  • Burst traffic from a specific tenant
  • A full cache
  • Partial failure of the prefill or decode pool
  • Router failover
  • Rolling upgrades
  • Concurrent handling of low-priority batch jobs and high-priority interactive traffic

Ultimately, llm-d’s value lies less in its peak throughput number than in how predictably it can control the latency and cost of important requests—even under congestion and during failures. Software Infra’s role is not to make GPUs work harder, but to ensure that limited GPU resources are used more consistently and according to policy.

Comments

Popular posts from this blog

Complete Guide to Apple Pay and Tmoney: From Setup to International Payments

The Beginning of the Mobile Transportation Card Revolution: What Is Apple Pay T-money? Transport card payments—now completed with just a single tap? Let’s explore how Apple Pay T-money is revolutionizing the way we move in our daily lives. Apple Pay T-money is an innovative service that perfectly integrates the traditional T-money card’s functions into the iOS ecosystem. At the heart of this system lies the “Express Mode,” allowing users to pay public transportation fares simply by tapping their smartphone—no need to unlock the device. Key Features and Benefits: Easy Top-Up : Instantly recharge using cards or accounts linked with Apple Pay. Auto Recharge : Automatically tops up a preset amount when the balance runs low. Various Payment Options : Supports Paymoney payments via QR codes and can be used internationally in 42 countries through the UnionPay system. Apple Pay T-money goes beyond being just a transport card—it introduces a new paradigm in mobil...

Cursor, Windsurf, Claude Code Compared: The Ultimate 2024 Guide to AI Coding Tools

AI Developer Tools: Cursor vs Windsurf vs Claude Code – What’s the Real Difference? With countless AI coding tools out there, which one should you choose? Cursor, Windsurf, Claude Code—on the surface, they might seem similar, but underneath lie fundamental differences. Let’s uncover the key distinctions among these three powerful tools. AI Model Accessibility: Direct vs Indirect Cursor offers direct access to Claude 4, excelling in complex code analysis. In contrast, Windsurf connects to AI models via API keys, while Claude Code integrates seamlessly as a VS Code plugin. These differences significantly impact how each tool operates and performs. Context Management: Manual vs Automated Cursor adopts a manual approach where developers control context themselves. Windsurf provides an automated context tracking system, and Claude Code automatically navigates and comprehends the entire codebase. Depending on your project’s scale and complexi...

New Job 'Ren' Revealed! Complete Overview of MapleStory Summer Update 2025

Summer 2025: The Rabbit Arrives — What the New MapleStory Job Ren Truly Signifies For countless MapleStory players eagerly awaiting the summer update, one rabbit has stolen the spotlight. But why has the arrival of 'Ren' caused a ripple far beyond just adding a new job? MapleStory’s summer 2025 update, titled "Assemble," introduces Ren—a fresh, rabbit-inspired job that breathes new life into the game community. Ren’s debut means much more than simply adding a new character. First, Ren reveals MapleStory’s long-term growth strategy. Adding new jobs not only enriches gameplay diversity but also offers fresh experiences to veteran players while attracting newcomers. The choice of a friendly, rabbit-themed character seems like a clear move to appeal to a broad age range. Second, the events and system enhancements launching alongside Ren promise to deepen MapleStory’s in-game ecosystem. Early registration events, training support programs, and a new skill system are d...