# Disaggregation vs. Chunked Prefill: Why Most Teams Should Stick With the Simpler Fix
The inference landscape in 2025 has been dominated by a single architectural pattern: splitting the prefill phase from the decode phase and running them on separate GPU pools. Major frameworks built the capability in. The community embraced it enthusiastically. The argument is compelling — prefill and decode have fundamentally different hardware demands, so separating them should unlock dramatic throughput gains.
For most production teams, however, that promise doesn’t hold up in practice. The gains that make disaggregation attractive at hyperscale disappear at the GPU counts most teams actually operate at. What’s more, a simpler technique built into mainstream serving frameworks handles the core problem disaggregation aims to solve — without any network overhead or infrastructure complexity.
## The Two Workloads That Share a GPU
Understanding why this tension exists requires looking at what prefill and decode actually do inside the GPU.
Prefill is a compute-heavy operation. The GPU processes the full input sequence through parallel matrix multiplications, pushing compute utilization to 80–95 percent on modern hardware. It’s the equivalent of a factory running at full capacity.
Decode is the opposite. It reads from the KV cache one token at a time, producing a single output token per step. Compute utilization on a top-tier GPU can drop below 5 percent during this phase. The GPU is mostly waiting on memory bandwidth, not crunching numbers.
When both workloads share the same GPU, they create a scheduling conflict. A large prefill request arriving mid-decode can inflate per-token latency by 2 to 30 times under bursty traffic patterns. The decode batch stalls while prefill hogs the compute units. This is the interference problem, and it’s real — but disaggregation isn’t the only way to solve it.
## Chunked Prefill: Solving Interference Without Adding Complexity
Chunked prefill breaks long input sequences into smaller segments and interleaves them with ongoing decode batches on the same GPU. Instead of dedicating separate hardware to each phase, it shares the GPU fairly between prefill chunks and decode steps.
The benefit is immediate and measurable: a 50 percent increase in total token throughput on standard serving frameworks with no architectural changes. There’s no network transfer involved, no separate node pool to manage, and no ratio to tune. The KV cache stays in GPU memory where it belongs.
Think of it like a toll booth where one oversized vehicle doesn’t shut down all lanes — it passes through one lane at a time while other traffic keeps flowing. Chunked prefill bounds the interference rather than eliminating it, and for workloads under roughly 50 requests per second with moderate prompt lengths, that bound is more than sufficient.
## The Hidden Costs of Splitting Prefill and Decode
When teams do choose disaggregation, they pay for it in three distinct ways that many write-ups overlook.
### The KV Cache Transfer Overhead
Once prefill and decode live on separate machines, every completed prefill must ship its KV cache over the network to the decode node. For a large model, that’s multiple gigabytes per request traversing the interconnect.
When both phases run on the same node, the KV cache stays in local GPU memory — zero-copy, near-instantaneous. The moment you split across nodes, that transfer hits whatever bandwidth your network fabric provides. Without InfiniBand or NVLink within the same rack, the networking bottleneck can erase any throughput gains from specialization.
The analogy is straightforward: you’ve separated two steps of a manufacturing process into different buildings. Each building can optimize its own work, but now you’re trucking half-finished products between them.
### The Operational Burden
Disaggregation doubles the infrastructure you need to manage. Separate node pools for prefill and decode each require their own scaling policies, health monitoring, and failure handling. The ratio of prefill to decode nodes depends on your specific workload mix, and that mix shifts over time — especially across days, weeks, and seasons.
There’s also no graceful fallback. If a prefill node goes down, the decode nodes cannot absorb its work, and vice versa. The roles are locked in at deployment time.
### The Silent Failure Cliff
Perhaps the most dangerous cost is that disaggregation can fail in ways that don’t surface as errors. Under high concurrency, systematic failures in KV cache transfer can return misleading responses to clients rather than raising exceptions. When input lengths exceed configured limits, a prefill node might abort a request and transfer a malformed KV state, causing the decode node to generate output from uninitialized memory — with no indication to the caller that something went wrong.
These failures don’t show up in low-concurrency testing. They emerge only when production traffic loads cross specific thresholds, making them difficult to catch before they reach real users.
## When Disaggregation Actually Makes Sense
Disaggregation isn’t universally wrong — it becomes the right call when three conditions align simultaneously.
First, you need enough GPUs to allocate clean integer ratios between prefill and decode pools. At small GPU counts (single digits to low tens), the possible ratios rarely match real workload demands. The rounding losses eat any throughput benefit.
Second, your network infrastructure must sustain the rate at which prefill nodes produce KV caches. If the interconnect is the bottleneck, decode nodes sit idle waiting for data, and you’ve simply moved the problem from compute contention to network saturation.
Third, your traffic mix must be stable enough that static node allocations remain sensible. A chat application with short prompts and long generations needs a very different prefill-to-decode balance than a retrieval-augmented generation pipeline with long prompts and short responses. If your workload shifts throughout the day, static pools leave you over-provisioned on one side and starved on the other.
For teams that don’t meet all three conditions simultaneously, chunked prefill remains the better default.
## How to Decide Before You Invest in Infrastructure
Before committing to any architectural change, measure the right signals at the right quantiles.
Focus on p95 time-per-output-token, not the median. The median hides the latency spikes that disaggregation is designed to address. If your p95 metric sits comfortably within your service-level objectives on colocated infrastructure with chunked prefill enabled, you likely don’t have the problem that disaggregation targets.
Profile the actual fraction of time-to-first-token spent in prefill. If prefill accounts for a small share of total latency, the bottleneck is elsewhere — queueing delays, scheduling contention, or network hops — and disaggregation won’t move the needle.
Finally, test KV transfer reliability at production concurrency levels, not just at low query-per-second rates. The failure modes in disaggregated systems only appear once traffic crosses specific concurrency thresholds, which means low-load testing gives a false sense of stability.
## FAQ
**What is the core difference between disaggregated serving and chunked prefill?**
Disaggregation runs prefill and decode on completely separate GPU pools, often across different machines, with KV cache transfers over the network between them. Chunked prefill interleaves prefill and decode work on the same GPU, sharing resources without any network hop.
**Does chunked prefill work for all model sizes?**
Yes. Chunked prefill is a scheduling technique that operates at the framework level and is independent of model size. It works equally well on small classification models and large language models with tens or hundreds of billions of parameters.
**Can I use both techniques together?**
In principle, yes — you could run chunked prefill within each node of a disaggregated architecture. In practice, the complexity rarely justifies the marginal benefit unless you’re operating at very large scale with specific workload patterns that demand both approaches.
**Why does disaggregation show such dramatic gains in research papers but not in production?**
Research benchmarks often use controlled workloads that are heavily prefill-biased and run at scales where GPU allocation is clean. Production workloads tend to be mixed, run at smaller GPU counts, and face real-world concurrency patterns that expose the hidden costs of disaggregation.
**Is there a GPU count threshold where disaggregation becomes worth it?**
Roughly speaking, the threshold is around 1,000 GPUs — where you can allocate integer node counts that genuinely match your traffic mix and where the KV transfer tax is amortized across enough throughput to justify the infrastructure complexity. Below that, chunked prefill is almost always the better choice.
**What should I measure to know if I have an interference problem?**
Start with p95 time-per-output-token under bursty production traffic. If the p95 sits within your SLO on chunked prefill, you don’t have an interference problem that disaggregation would solve. Also check whether prefill execution time is a meaningful fraction of total time-to-first-token — if it isn’t, the bottleneck is elsewhere.
## Conclusion
The default starting point for almost every team should be chunked prefill. It solves the scheduling interference that creates latency spikes in most production workloads, requires zero infrastructure changes, and is available today in mainstream serving frameworks.
Disaggregation is a legitimate architecture for a narrow set of conditions: very large GPU clusters, high-speed interconnects, stable traffic patterns, and the engineering bandwidth to manage P:D ratio tuning and KV transfer reliability. That description fits hyperscalers and large inference providers. It does not fit the majority of teams shipping LLM-powered features in production today.
Benchmark your p95 TPOT with chunked prefill enabled. If your latency targets hold, there is no reason to add the complexity and risk of disaggregation. Start simple, measure rigorously, and escalate your architecture only when the data demands it.
Thank you for reading
