# SoL-Pi: How Auto-Research Found Four Harness Optimizations That Cut Token Costs for Open-Source Coding Agents by Over a Third
The landscape of AI-powered software development has shifted dramatically. Coding agents now run for extended periods — sometimes hours — rather than mere minutes. Every file edit, every test execution, and every log read flows back into the model’s working context. This creates a significant challenge: as tasks grow in complexity, the volume of tokens consumed skyrockets, driving up both latency and cost.
A group of researchers from NVIDIA, Nanyang Technological University, and the Massachusetts Institute of Technology set out to tackle this problem head-on. Their work introduced SoL-Pi, a collection of four efficiency mechanisms designed for the open-source Pi coding agent. What makes this effort particularly notable is the method used to discover these optimizations: an AI-driven auto-research loop that operated at the harness layer, the infrastructure that manages tool calls, context tracking, observations, and task delegation.
## The Problem With Manual Optimization
Most approaches to reducing the cost of running coding agents focus on lowering the price per individual token — through faster hardware kernels, aggressive quantization, or switching to cheaper models. SoL-Pi takes a fundamentally different angle. Instead of making each token cheaper, it reduces the total number of tokens a given task requires in the first place.
The harness layer is where this strategy pays off the most. This is the middleware responsible for orchestrating how an agent interacts with tools, manages growing context windows, processes tool outputs, and delegates work across multiple sub-agents. Manually tuning this layer is painstaking and fraught with trade-offs, since improvements in one area can inadvertently increase costs downstream.
Automated systems like Meta-Harness have attempted to tackle this problem, but a well-known pitfall is overfitting: harness configurations optimized for a specific set of benchmark tasks often fail to generalize to unseen workloads.
## A Self-Driving Search Process
To avoid these limitations, the team built a fully automated discovery pipeline. A research AI monitored execution traces from a separate agent running the base Pi system. It then proposed modifications to the harness, tested each proposal in isolated environments, and evaluated whether the changes held up.
The scope of this search was enormous:
– **152 proposed optimization directions**, spread across six distinct categories including context management, progress tracking, tool handling, delegation strategies, prompt and policy design, and general improvement and evaluation methods.
– **535 executable environments** — 495 of which were constructed from real-world GitHub issue and pull request pairs, and 40 of which were synthetic tasks equipped with automated verifiers.
– **Over 3,000 individual search runs**, encompassing more than 60,000 agent-environment interactions.
Each search cycle was self-contained and disposable. It followed an autoresearch pattern augmented with a structured implementation step and an independent review pass. Crucially, the acceptance rules were locked in before the search began. The optimizer could not alter these guardrails. Every candidate modification had to satisfy two conditions simultaneously: all capability metrics needed to remain within a predetermined tolerance, and at least one efficiency metric had to show a measurable improvement.
The EdgeBench benchmark, comprising 51 public tasks, was held completely separate from the search. Eleven of those tasks served as a one-way gate for accepting frozen candidates, while the remaining forty were reserved for final evaluation. Results from the held-out tasks never fed back into the search loop, ensuring an honest measurement of generalization.
## The Four Mechanisms That Survived Selection
After sifting through hundreds of proposals, exactly four mechanisms emerged as durable winners. Each one targets a different source of token waste in the coding agent pipeline.
### 1. Action Fusion
A typical coding agent workflow involves editing a file and then issuing a separate command to test, build, or run it. These two steps usually require two distinct API round trips — one for the edit and another for the agent to read the result and decide on the next command.
Action Fusion collapses both into a single tool request. The harness attaches the follow-up command directly to the file edit and returns both outcomes in a unified observation. This eliminates an entire model round trip for each paired edit-and-verify cycle.
Across the EdgeBench evaluation, roughly three-quarters of all tasks trigger this mechanism when running on the GPT-5.6 Sol backend. On average, each triggered task sees Action Fusion activate over seventy times, yielding a token efficiency gain of nearly a quarter on those tasks alone.
### 2. Online Context Compact
As a coding agent works through a multi-step plan, the context window fills with previous observations, tool outputs, and conversation history. Eventually, the model either hits the context limit or wastes tokens on material that is no longer immediately relevant.
Online Context Compact solves this by tracking plan progression through a dedicated state mechanism. At the moment each plan step finishes, the harness estimates how many further requests are likely needed. It then weighs the projected input savings from compressing older context against the overhead cost of rewriting the prompt cache for the model. If the savings outweigh the cost — or if the context is dangerously close to the window limit — the harness triggers the model’s native compaction feature. Otherwise, it leaves the full history intact.
This dynamic gating ensures that compaction happens only when it is genuinely beneficial, avoiding the pitfalls of premature or unnecessary context pruning.
### 3. ObservationPack
Tool outputs — especially build logs, test results, and large file diffs — can easily exceed 10 KiB in size. When the same bulky output needs to be referenced again in subsequent steps, retransmitting it in full burns through tokens rapidly.
ObservationPack archives large tool outputs locally and sends them in full during the next two provider requests. Starting from the third request onward, the model receives a stable handle, the original payload size, and a short excerpt showing the head and tail lines of the output. The complete, exact content remains retrievable on demand through the handle.
This approach dramatically cuts repeated transmission of large logs while ensuring that no evidence is ever truly lost. The model can always fetch the full original when precision matters.
### 4. Evidence-Preserving Reducer
Build and test logs often span many kilobytes. Feeding entire logs back to an expensive frontier model on every subsequent step is wasteful — especially since the agent’s main job is usually diagnosis and action selection, not raw log reading.
The Evidence-Preserving Reducer routes logs of at least 4 KiB from a predefined set of commands to a cheaper secondary model, which produces a compact structured receipt. A deterministic verifier then checks the receipt across five dimensions: schema correctness, source hash integrity, exit status accuracy, exact quote preservation, and size reduction.
If verification passes, the harness stores the receipt and lets the main agent continue working with the compact summary. File reads and search results bypass the reducer entirely. In three specific scenarios — verification failure, suspected credential exposure, or a receipt that is not actually smaller than the original — the system falls back to transmitting the complete log.
## Deployment and Compatibility
SoL-Pi is designed to integrate smoothly into existing workflows. It ships on GitHub under an MIT license as an extension that runs on an unmodified release of the Pi coding agent. The implementation has been tested with Pi version 0.85.1 and requires Node.js version 22.19 or newer. No modifications to the base Pi installation are necessary — everything operates as an opt-in overlay.
## Measured Impact
On the EdgeBench evaluation, the full four-mechanism stack delivered substantial reductions in both token traffic and API expenditure:
– **Token traffic was cut by 44.7% to 49.0%** compared to the base Pi agent, depending on the backend used.
– **API costs dropped by approximately 33%** across the evaluated task set.
– Performance scores remained close to the baseline Pi agent on both GPT-5.6 Sol and Opus 5 backends, confirming that the efficiency gains did not come at the expense of task completion quality.
Beyond the headline EdgeBench numbers, SoL-Pi also demonstrated strong results in other benchmarks. On Terminal-Bench 4, the full stack solved 15 tasks compared to Pi’s 18, while slashing total cost by 26.3%. For individual International Mathematical Olympiad problems, the system achieved a per-passed cost of just $20.90 — the lowest among three compared harness configurations. In swarm scenarios involving 20 Pi workers, API cost dropped by 26.8%, with total cycles falling from 1,366 to 1,127.
## Frequently Asked Questions
**What is SoL-Pi?**
SoL-Pi is a set of four efficiency mechanisms for the Pi open-source coding agent. It was discovered through an automated AI research loop that tested hundreds of harness-level modifications across thousands of environments and task runs.
**What does the harness layer do?**
The harness is the middleware that manages how a coding agent calls tools, tracks context across multi-step workflows, processes observations from tool outputs, and delegates work to sub-agents or subsequent reasoning steps.
**How does SoL-Pi differ from other efficiency approaches?**
Rather than reducing the cost per individual token through quantization or cheaper models, SoL-Pi reduces the total number of tokens each task consumes by optimizing how the harness orchestrates tool calls, manages context windows, and compresses large outputs.
**Can SoL-Pi degrade task performance?**
The search process enforced strict capability guardrails throughout. Every candidate had to maintain all capability metrics within a predefined tolerance while also improving at least one efficiency metric. Final evaluations confirmed that SoL-Pi’s scores remain close to the baseline Pi agent on both GPT-5.6 Sol and Opus 5 backends.
**How was the search process validated?**
The EdgeBench benchmark, containing 51 public tasks, was held out entirely from the search. Eleven tasks were used as a one-time acceptance gate for frozen candidates, and forty were used for final evaluation. No results from the held-out tasks were ever fed back into the search loop.
**Is SoL-Pi easy to install?**
Yes. It is distributed as an MIT-licensed extension on GitHub and integrates with an unmodified Pi release. It requires Node.js 22.19 or newer and Pi version 0.85.1.
**What is the Action Fusion mechanism?**
Action Fusion merges a file edit and its subsequent test, build, or run command into a single tool request, eliminating an extra API round trip that the base agent would otherwise require.
**How does the Evidence-Preserving Reducer ensure correctness?**
It uses a deterministic verifier that checks the compact receipt against five criteria — schema validity, source hash, exit status, exact quote preservation, and size reduction — before accepting it. If any check fails, the system falls back to the original log.
## Conclusion
SoL-Pi demonstrates that automated research at the harness layer can uncover efficiency improvements that are both substantial and generalizable. By reducing token traffic by nearly half and cutting API costs by roughly a third — without sacrificing task performance — it offers a practical path forward for teams running coding agents at scale. The use of an AI-driven search loop, combined with strict held-out evaluation and locked acceptance rules, provides a blueprint for how future efficiency research can be conducted rigorously and reproducibly. As coding agents continue to take on longer and more complex development workflows, innovations like SoL-Pi will become increasingly essential for keeping costs manageable and performance reliable.
Thank you for reading
