**Article: A Deterministic Prompt Pruner for Long‑Running LLM Conversations**
Long‑running agent conversations suffer from a familiar problem: the prompt grows without bound. Over time, chat history, stale tool outputs, duplicated retrieved passages, and outdated context accumulate, inflating token count, increasing cost and latency, and potentially degrading reasoning quality.
A common response is positional truncation, but this is fragile—it can discard critical, older information that later turns still depend on. Instead, a principled pruning mechanism is needed, one that understands dependencies and guarantees correctness.
**Design Principles**
The solution is a deterministic prompt pruner with three sequential passes:
1. **Expired Context Elimination**: Removes outdated tool outputs; only the most recent execution under a given key is retained.
2. **Duplicate Context Elimination**: Normalizes and deduplies retrieved documents, keeping the first occurrence and dropping repeats.
3. **Dependency Restoration**: Ensures nothing a later message references is accidentally removed. If a kept message references a definition that was dropped, the definition is restored.
Crucially, the pruner is fully deterministic—no embeddings or LLM calls are involved—and idempotent: pruning an already‑pruned prompt produces no further changes.
**Bug-Driven Improvements**
Early benchmarks revealed subtle flaws:
– Synthetic corpora with fixed duplicates masked real behavior, causing measured reduction to shrink as conversations grew.
– Dependency restoration was initially untested because corpora never created scenarios where a definition was legitimately removed.
Fixing these issues showed immediate value. Once tool outputs were allowed to define dependencies, restoration events increased from zero to dozens, while 100% of required facts were preserved across all tests.
**Benchmark Results**
The pruner was evaluated across three realistic workloads:
– **Plain chat** (minimal retrieval, occasional tools): 2–4 % token reduction.
– **RAG assistant** (retrieval every turn): 27–32 % reduction, driven mainly by duplicate elimination.
– **Tool‑heavy agent** (frequent tools and retrieval): 33–34 % reduction.
Across 15 configurations (3 workloads × 5 conversation sizes up to 2,000 turns and 131k tokens):
– All required facts were preserved.
– Every run reached a stable fixed point.
– Overhead remained under 50 ms even at the largest scale.
**Why This Matters**
Unlike truncation, dependency‑aware pruning prevents silent breakage of conversation chains. It balances compression with correctness, offering substantial token savings without risking lost context.
**Integration**
The pruner sits between conversation state collection and LLM prompt construction. It can be run on every turn safely, thanks to idempotence. Integration requires defining `DEFINE` and `REF` markers—an area where production systems can use structured metadata instead of raw content tags.
**Limitations and Future Work**
– Dependency detection is syntactic, not semantic. Paraphrased references are not recovered.
– Benchmarks are synthetic; applying the same methodology to production traces is the next step.
– The approach complements rather than replaces semantic compression techniques (e.g., embeddings or learned compressors), which can be added as a downstream step.
**Bottom Line**
A small, deterministic pruning layer can meaningfully reduce prompt bloat while guaranteeing that critical information is retained. For long‑running agents, predictable, correct context management is more valuable than raw compression alone.
—
**Source:** Adapted from “I built a deterministic pipeline that prunes redundant conversation state before the prompt ever reaches the model…” (https://contributor.insightmediagroup.io/o/prompt-pruner).
