# Understanding llm-d 1.0: Google’s Open-Source Inference Engine for Large-Scale AI Serving
## What Is llm-d?
Large language models are powerful, but serving them efficiently at scale is a major engineering challenge. When thousands of users send requests simultaneously, traditional load-balancing methods like round-robin often fall short — especially for agent-based workflows where context carries across multiple steps of a conversation.
llm-d is a distributed inference stack designed to solve this problem. Think of it as a smart traffic controller that sits between users and groups of AI model servers. Instead of sending every request to a random server, it keeps track of which server already holds the relevant conversation history and routes accordingly. This dramatically reduces wasted computation and speeds up responses.
The project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA. Today, it has over 500 contributors from more than 25 organizations and is governed under the Cloud Native Computing Foundation as a Sandbox project. It runs on Kubernetes and supports NVIDIA, AMD, and Intel GPUs as well as Google’s own TPUs. It works with two of the most popular open-source model serving engines: vLLM and SGLang.
The latest release, version 1.0, marks the first time this software has been deemed stable enough for production use. That means the APIs are locked in, the core functionality is battle-tested, and teams can rely on it for their day-to-day operations.
## What Makes Version 1.0 Different
Earlier versions of llm-d focused on proving that the core ideas worked. Version 1.0 is about making those ideas reliable, manageable, and ready for real-world environments.
**Multi-tenant quality of service** is a standout addition. In shared deployments, a sudden spike from one user or application can starve others of resources. The new flow-control system prioritizes latency-sensitive requests and ensures fair distribution of capacity.
**Zero-downtime upgrades** are another critical improvement. Through a feature called DisaggregatedSet, teams can roll out new versions of both the prefill and decode processing pools simultaneously without dropping active sessions. A hot-standby router ensures that maintenance or upgrades never interrupt traffic.
For teams running agent-based applications, the **agent-aware serving** layer introduces session-aware routing, turn-priority scheduling, and native support for the Responses API. This means an agent that sends multiple turns in a multi-step task can reuse cached context, avoiding redundant computation.
**Async inference** is also new. The llm-d-async-processor can pull jobs from message queues like Google Cloud Pub/Sub or Redis and process them during idle capacity. This is particularly useful for background tasks, batch jobs, and reinforcement learning workloads where timing is less critical.
Finally, **TorchTPU support** expands the range of hardware options. PyTorch models can now run natively on TPUs through upstream vLLM and SGLang, giving teams more flexibility in how they build their inference infrastructure.
## How the Routing System Works
At the heart of llm-d is a router that sits in front of model server pools. It uses the Kubernetes Gateway API Inference Extension to manage traffic. When a request arrives, the router checks a registry of which replicas currently hold cached parts of the conversation. It then considers current load and predicted latency to pick the best replica.
This is especially important for agent systems. An agent might send a user message, receive a response, send a follow-up, and so on. Each step benefits from landing on the same server that holds the previous steps’ context. Without this, every new message would force the model to re-process earlier turns, wasting time and compute.
When GPU memory runs low, the system offloads cached key-value data in tiers — first to host CPU memory, then to local SSD storage, and finally to distributed file systems like Google Cloud Managed Lustre. This tiered approach ensures that longer conversations don’t cause evictions or crashes. The system supports offloading connectors from vLLM, as well as tools like Mooncake and HiCache from SGLang.
Separating prefill and decode operations into distinct pools is another architectural choice worth noting. Prefill — the initial processing of a new prompt — is computationally heavy. Decode — generating each new token — is more memory-bound. By running them on different resource pools, each type of work gets the hardware it needs most.
## What the Benchmarks Show
All benchmark figures below come from Google and early adopters of the technology.
On Google’s Ironwood TPU platform, which uses the latest TPU generation, prefix-aware routing combined with host-memory offload reduced median time-to-first-token by more than 85%. Overall throughput nearly doubled compared to running the same models behind a plain Kubernetes service.
Google’s own Gemini platform saw its prefix cache hit rate double, rising from roughly 35% to 70%. This translated to a more than 35% improvement in time-to-first-token for coding agents and over a 50% reduction at the 95th percentile during bursty chat workloads.
A large-scale test spanning 17,000 nodes across three Google Kubernetes Engine clusters achieved 2.9 times higher throughput while maintaining 99.9% success rates and less than 1% overhead.
For a 70-billion-parameter model running on an A3 Mega instance, offloading to Lustre and CPU RAM improved time-to-first-token by about 40% on very long 50,000-token prompts. End-to-end latency dropped 30%, and GPU hours used fell by nearly 60%.
In reinforcement learning scenarios, time-slicing — which interleaves inference work with training cycles — lifted accelerator utilization from roughly 40% to above 70%, cutting wasted idle time significantly.
Early adopters outside Google report similar gains. One company reported prefix cache hit rates climbing to 75–80%. Another, working with TPUs, measured over four times throughput, pushing daily token volume from 1.47 billion to 5.97 billion per 1,000 chips.
## How Does It Compare to Alternatives?
Two other prominent projects in this space are NVIDIA Dynamo and AIBrix. While they share some goals, each has a distinct approach.
NVIDIA Dynamo is led by NVIDIA and has been around longer, with a larger community on GitHub. It supports more inference engines, including TensorRT-LLM, and focuses heavily on NVIDIA’s own GPUs.
AIBrix originated at ByteDance and is now part of the vLLM organization. It emphasizes heterogeneous GPU support and distributed caching.
llm-d differentiates itself through its broad hardware support — NVIDIA, AMD, Intel GPUs, and Google TPUs — and its governance under the CNCF, which emphasizes neutrality and broad industry participation. It is also the only one of the three built natively on Kubernetes Gateway API extensions and designed from the ground up for agent and RL workloads.
Importantly, these tools are not strictly competitors. Some organizations run llm-d alongside Dynamo, and NVIDIA actively contributes components to the llm-d project. The ecosystem is still young, and overlap often means more choices and faster innovation for users.
## Deploying llm-d
On Google Cloud, the easiest path is through GKE Inference Gateway, which can run the llm-d router as a managed service. This setup supports multi-cluster and multi-region deployments with global failover.
Scaling is handled by Kubernetes-native tools: KEDA and Horizontal Pod Autoscaler adjust the number of model replicas based on queue depth and predicted latency. Kueue manages per-model quotas to prevent any single team from consuming all available resources.
Google recommends a phased rollout. Start by placing prefix-aware routing in front of an existing vLLM deployment. Once that is stable, add flow control for quality of service, then introduce tiered KV offload and disaggregation. Teams can compare different configurations and performance profiles using a benchmarking platform called Prism.
Helm charts are available for verified deployments of both TPU and GPU pools behind a single endpoint, simplifying setup for most common configurations.
—
## Frequently Asked Questions
**What does llm-d stand for?**
The name stands for “LLM Dispatcher” or “distributed LLM” serving, referring to its role in coordinating model inference across many machines.
**Do I need Google Cloud to use llm-d?**
No. llm-d runs on Kubernetes and is cloud-agnostic. It supports on-premises clusters, other cloud providers, and hybrid setups. Google Cloud simply offers one of the most polished integration paths today.
**Which model serving engines does llm-d support?**
Currently, vLLM and SGLang. Both are popular open-source engines that handle the actual model inference work.
**Is llm-d only for large enterprises?**
Not exclusively. While the largest benchmarks involve thousands of nodes, the software can scale down to smaller deployments. It is designed for any organization that needs to serve many users or agents from a shared pool of GPU or TPU resources.
**What is prefix-aware routing?**
It is a technique where the router keeps track of which model server already has part of a conversation’s history cached. By sending new requests to that same server, the system avoids recomputing shared context, saving both time and compute.
**How does tiered KV cache offload work?**
When a GPU runs low on memory, the system moves cached key-value data to the next tier in a hierarchy: CPU RAM, local SSD, and then a distributed file system. This prevents the model from having to discard useful context and recompute it later.
**What is DisaggregatedSet?**
It is a Kubernetes feature within llm-d that allows prefill and decode server pools to be upgraded together as a single unit, without dropping active user sessions. This is key for maintaining uptime during software updates.
**Can llm-d handle reinforcement learning workloads?**
Yes. It includes time-slicing capabilities that let inference jobs share accelerators with training workloads, boosting overall hardware utilization.
**Is llm-d free to use?**
Yes. It is open-source software released under the Apache 2.0 license, which permits commercial use, modification, and redistribution.
**Who maintains llm-d?**
Red Hat, IBM, and Google are the lead maintainers. The project has more than 500 contributors and is accepted under the CNCF Sandbox program.
—
## Conclusion
llm-d 1.0 represents a meaningful step forward for organizations that need to serve large language models efficiently at scale. By combining cache-aware routing, tiered memory offloading, disaggregated prefill and decode processing, and agent-aware session management, it addresses many of the pain points that arise when running model-serving infrastructure across large Kubernetes clusters.
The benchmarks — while self-reported — paint a compelling picture: faster response times, higher throughput, better hardware utilization, and improved reliability during upgrades. For teams already invested in the Kubernetes ecosystem and looking to run vLLM or SGLang on diverse hardware, llm-d offers a well-integrated, production-ready foundation.
As the project continues to grow under CNCF governance with broad industry participation, it will be worth watching how it evolves and how the broader community adopts it alongside competing tools like NVIDIA Dynamo and AIBrix.
Thank you for reading
