# Engineering Long-Running Agents: The Hidden Layer That Prevents Context Collapse
In its most basic form, an agent is an LLM invoking tools in a repetitive cycle. This loop handles brief tasks with ease. However, when a job stretches to an hour and requires hundreds of tool invocations, the system collapses in two predictable ways: the context overflows, and the agent loses sight of its primary objective.
The fix lies not in the language model itself, but in the surrounding infrastructure—often called the harness—which manages everything except the model’s core reasoning. This piece explores the machinery that transforms a basic loop into a deep, resilient agent, examining four key mechanisms that keep long-running tasks on track.
## The Illusion of Infinite Context
Expanding the context window seems like the obvious solution, but empirical evidence suggests it offers diminishing returns. Attention mechanisms create quadratic pairwise relationships for every token added, meaning each new input drains a finite “attention budget.” Context is a resource subject to diminishing returns, not an infinite bucket.
In an agent loop, the problem compounds. A typical workflow might require dozens of tool calls, and each observation accumulates in the context window, remaining there until the task concludes. The original instruction gradually drifts toward the center of the window, precisely where recall degrades. Without active management, losing the objective on long tasks is an expected mathematical outcome, not just a software bug. The input-to-output token ratio in agent loops can be extreme, with the original prompt becoming indistinguishable from the noise of intermediate tool results.
## Mechanism 1: Context Budgeting and Offloading
The first responsibility of a robust harness is deciding what should never enter the model’s active context at all. One prominent framework implements a two-tiered offloading system with hard numeric thresholds. When a tool response exceeds 20,000 tokens, the harness writes the raw data to the local filesystem and replaces it in the window with a file path plus a brief preview of the opening lines.
When the session context hits 85% of the model’s maximum window, older write and edit actions—whose full file contents already exist on disk—are truncated to lightweight pointers. Only after exhausting these offloading options does the system fall back to summarization.
This budgeting also applies to what loads before the first prompt. Leading implementations cap preloaded memory at a few hundred lines or a strict kilobyte limit, and defer tool schemas by default, only loading full schemas on demand. An architecturally powerful variation of this is delegation: spawning isolated sub-agents to explore specific domains. The sub-agent burns tokens exploring within its own bounded window, but returns only a distilled summary—often just a few hundred tokens—to the parent, ensuring the main context never gets polluted with raw exploration data.
## Mechanism 2: Compaction and Structured Summarization
When offloading isn’t sufficient, the harness must summarize. Compaction involves taking a conversation nearing its limit, distilling it, and reinitializing the context with that summary. This is precisely where losing the objective most frequently occurs, as a lossy summary might accidentally discard a critical constraint buried in the history.
Modern approaches treat compaction structurally rather than as a generic text reduction. Instead of a free-form summary, some systems generate a document with dedicated fields for session intent, created artifacts, and subsequent actions. Crucially, the complete original transcript is also saved to the filesystem, allowing the agent to retrieve a fact that the summary omitted by searching the file later.
At the API level, server-side compaction is now available, allowing developers to pass a compacted context window back into the model seamlessly. Furthermore, engineers can inject custom instructions immediately after the compaction process, ensuring specific constraints survive the reset. When writing custom instructions for this step, they replace the default prompt entirely, making the compaction prompt a first-class engineering artifact rather than a mere setting.
## Mechanism 3: Todo-State and Continuous Recitation
While compaction protects the goal at the moment of summarization, continuous recitation protects it between turns. A common technique involves maintaining a mutable checklist within the agent loop. The agent creates a task list and rewrites it step-by-step as it progresses, checking items off. By appending this list to the end of the context window on a regular interval, the global plan is pushed into the model’s most recent attention span, significantly reducing middle-drift. No architectural change is required; it is simply natural language used to bias the model’s own attention.
However, this technique is not a guaranteed win. Recent evaluations across various task categories have shown that removing this mechanism can sometimes yield slightly better performance and lower cost, as the act of rewriting the list consumes tokens on every turn. It remains most valuable for extended, multi-step operations or when guiding less capable models. The underlying principle is that the goal should exist as a frequently updated, mutable artifact, not merely as a fading message in history.
## Mechanism 4: Cross-Session Memory Strategy
The final component is what persists after a task finishes. Persistent memory allows a system to recall prior work without re-researching from scratch. Systems store conversation events and run extraction strategies in the background, so a coordinator can query a recall tool on the next run rather than sifting through raw logs.
However, persistent context is not free. Research has found that auto-generated repository context files can raise inference costs by over 20% without reliably improving task success rates. Therefore, it is best practice to keep these memory files highly concise—ideally under a few hundred lines—and to move detailed reference material into modular, on-demand components that only load when specifically needed. Without at least one configured extraction strategy, raw events are stored but nothing is actually extracted for future retrieval, wasting the persistence mechanism entirely.
## FAQ
**Q: Why can’t we just give the model a larger context window to fix agent failures?**
A: Attention creates quadratic relationships for every token, meaning each added token depletes a finite attention budget. In agent loops, tool results pile up and push the original instruction toward the middle of the window, where recall degrades. A larger window delays the problem but does not solve the underlying management issue.
**Q: What happens when offloading and summarization both fail?**
A: If the context reaches 100% capacity without intervention, the system suffers an overflow. In a real loop, the earliest messages, including the original instruction, are the first to be truncated or ignored by the model, usually causing the agent to abandon the task or hallucinate.
**Q: Is maintaining a running task list always beneficial for agents?**
A: No, it adds a per-turn token cost. Recent evaluations suggest that for certain tasks, disabling the task list can yield slightly better reward and lower cost. It is recommended primarily for long, multi-step tasks, less capable models, or interfaces that need to display progress to a human.
**Q: How does cross-session memory impact inference cost?**
A: It imposes a standing tax on the attention budget. Auto-generated memory files have been shown to increase inference costs significantly (by 20% or more) across benchmarks. Memory should be kept minimal and strictly scoped to avoid paying this tax unnecessarily on every new session.
## Conclusion
The transition from a simple tool-calling loop to a resilient, long-running agent requires rethinking the entire architecture. The model alone cannot solve context decay; the harness must actively budget, offload, compact, and recite the objective. By treating context as a finite resource—budgeting what enters, summarizing what lingers, and externalizing what is needed—developers can build agents capable of sustained, complex work across hours and hundreds of steps. Thank you for reading
