# A New Approach to Cutting LLM Agent Costs: KV Cache Engineering That Lets Agents Manage Their Own Memory
### The Problem With Trimming Agent Conversations
Large language model agents carry on long conversations across multiple turns. Each round adds tool calls, results, and reasoning steps to the context window. Over time, this accumulates into thousands of tokens that must be processed at every step — and the cost of processing this stored context, particularly the KV cache, can dominate overall expenses.
Most existing solutions tackle this by summarizing or trimming the text history. But this approach has a hidden cost: when you edit the conversation prefix, every cached state after that edit becomes invalid. The system must then recompute (“re-prefill”) all the retained text from scratch. The result is wasted computation and lost information from middle rounds that the summary may overlook.
A research effort from two leading Chinese universities has introduced a fundamentally different strategy — one that leaves the conversation text untouched and instead manipulates what the model can actually “see” in its memory.
—
## What Is KV Cache Engineering?
The idea is simple but powerful. Rather than rewriting an agent’s conversation history, a cache engineering system selectively removes or reorganizes the internal KV cache entries that power the model’s attention mechanism. Messages are still present in the conversation — nothing is deleted from the text — but certain past entries are hidden from attention so they no longer consume compute resources.
Think of it like cleaning your desk without throwing away any documents. You simply slide some folders out of the way so they don’t take up space while you work, but they remain available if you need them later.
This is different from traditional context engineering in a critical way: because the underlying text doesn’t change, none of the cached computations become invalid. Retained entries continue to be reused without any recomputation at all.
—
## How It Works: Three Cache Operations
The system defines three core operations for managing the KV cache:
**Append** — the standard behavior when new tokens are generated. Every new turn adds entries to the cache as normal.
**Drop** — selected messages are removed from the attention computation. The tokens still exist in the conversation, but the attention mechanism simply skips them. This frees up memory and reduces the work the model needs to do.
**Reposition (Repos)** — this is where the approach gets clever. After some entries are dropped, gaps appear in the sequence of cached positions. Repos rotates the keys of retained tokens into a compact, contiguous set of positions starting from zero. The values remain untouched, which means no hidden states are recomputed. This tiny operation produces a surprisingly large boost in quality because the model can still attend to older information without the overhead of gaps in its cache.
In testing, reposition alone improved BrowseComp-Plus performance from 55% to 62% for one of the models evaluated.
—
## How Agents Decide What to Drop
The system includes a policy library that determines which cache entries get removed and when. Policies can be rule-based — for example, always keeping the most recent 12 tool responses — or they can be token-count or turn-count based. There is also a tool-based policy that lets the model itself call a `drop_context` function during its reasoning, giving the agent direct control over its own memory.
In one experiment, a model was able to refine its own cache policy through iteration. After practicing on a set of tasks, it learned to drop strategically, achieving cost savings beyond what a human-designed policy could deliver.
—
## Performance Across Models and Benchmarks
The system was tested on three models of varying sizes:
– **A large open-source model** with 120 billion total parameters but only 5.1 billion active at any time.
– **A mid-range model** with 230 billion total parameters and 10 billion active.
– **A compact model** with 30 billion total parameters and 3 billion active.
Four agent benchmarks were used: BrowseComp-Plus, GAIA-Text, SWE-bench Verified, and xbench-DeepSearch-2510.
Across all models and benchmarks, the average cost dropped by **27.7%** compared to summarization-based baselines. At the same time, task scores **improved by an average of 1.7 points**. The largest gains were seen on the long-context BrowseComp-Plus benchmark, where the system scored 37% at 96K tokens versus 24% for the baseline summarization approach.
One notable case: on a difficult coding benchmark, a model went from costing 20 cents per task to just 9.7 cents while also improving from a score of 40.0 to 43.2.
—
## Serving Speed Matters Too
Beyond accuracy and cost, the system also improves raw throughput. In serving benchmarks with high concurrency — when many requests arrive simultaneously — the system handled requests up to **1.5x faster** and delivered output up to **1.2x faster** than the summarization baseline.
At the highest concurrency level tested, P95 latency dropped from over 3 minutes down to roughly 30 seconds per request. Per decode step, full-attention computation time was reduced by more than half. The reason is straightforward: without the burden of re-prefilling entire conversation histories, the model spends less time on cache management and more time generating useful output.
—
## How It Compares to Other Approaches
Several alternative KV cache methods exist, each with different tradeoffs:
| Approach | What It Does | Best For |
|—|—|—|
| **KV Cache Engineering** | Agent-directed dropping and repositioning of cache entries | Multi-turn tool-calling agents |
| **Text Summarization** | Trims conversation history by rewriting it | General agent harnesses |
| **Prefix Reuse Methods** | Shares common prefixes across requests | General serving workloads |
| **Fixed Window Methods** | Keeps a sliding window of recent tokens | Streaming dialogue |
| **Attention-Based Pruning** | Removes low-attention positions from prompts | Long single prompts |
The cache engineering approach is unique in that it is specifically designed for agents that call tools repeatedly over many turns. Unlike fixed-window methods, it does not blindly discard older information — it uses a policy to decide what to keep. Unlike summarization, it never has to recompute what it has already computed.
—
## Getting Started
The software is open-source under a permissive license and integrates with a popular serving framework for language models. It runs on standard NVIDIA GPUs and can serve even the largest models on as few as two GPUs using tensor parallelism. The API follows standard conventions, so existing agent codebases can adopt it with minimal changes — just two optional fields are added to the request payload to specify which messages to drop and where to reposition the cache.
—
## FAQ
**Q: Does dropping cache entries cause the model to forget information?**
A: Not necessarily. Dropped messages remain in the conversation history, but the model cannot attend to them directly. In practice, the most recent and most relevant tool responses are kept, while older ones are pruned. The reposition operation further ensures that retained entries remain accessible.
**Q: Is this technique limited to specific model architectures?**
A: Current support covers models from a small number of families (Qwen, GPT-OSS, and MiniMax-M2). Work is ongoing to extend the approach to other architectures, including those that use linear attention mechanisms.
**Q: Can this be used with cloud-hosted API models, or only self-hosted deployments?**
A: The system is designed for self-hosted deployment. You run the serving software on your own GPU infrastructure. It is not a plug-in for third-party API providers.
**Q: How do the policies decide what to drop?**
A: Built-in policies include turn-based (drop after a certain number of turns), token-based (drop when the cache exceeds a size threshold), and model-based (let the model choose via a tool call). Custom policies can also be defined.
**Q: Does the reposition operation add significant computational overhead?**
A: Very little. Repos only rewrites the positional encodings (keys) using a rotation operation. The values are left untouched, so no hidden state recomputation is needed. The overhead is negligible compared to the savings from avoiding re-prefill.
**Q: Are there situations where cache engineering performs worse than summarization?**
A: Yes. On certain benchmarks with certain models, the accuracy can fall slightly below summarization-based approaches. This is why the policy design and drop strategy matter — not all dropped content is equally safe to remove. Reasoning cache in particular, when dropped, can reduce accuracy on coding tasks.
—
## Conclusion
Managing the memory of long-running LLM agents is one of the most practical bottlenecks in deploying these systems at scale. Traditional approaches of trimming text history come with a steep recomputation cost and risk losing important middle-round details. Cache engineering offers a fundamentally different path: leave the conversation intact, but let the model decide what it no longer needs to hold in active memory.
The results are compelling — lower costs, higher accuracy on long runs, and faster serving under load — all without requiring any model fine-tuning. As agent systems grow more complex and handle ever-longer workflows, techniques like this one will become essential infrastructure for anyone building or deploying LLM agents in production.
Thank you for reading
