# Can AI Models Build and Evolve Their Own Agent Harnesses? A Deep Dive into the New Evaluation Paradigm
## Introduction
When we evaluate large language models, we typically focus on what they produce — an answer, a code snippet, a reasoning chain. But a growing body of research suggests that what truly determines real-world performance is not the final output itself, but the environment in which the model operates: the execution loop, the tools it can access, the way it manages context and state, and how it recovers from failures. This surrounding infrastructure is known as an **agent harness**, and it turns out to be enormously influential.
Consider this striking data point: under one particular harness, a leading model solves roughly 35% of tasks, but under a different harness with identical weights, that same model clears nearly 50%. The model hasn’t changed — the harness has. This sensitivity raises a fundamental question: if the harness is so important, why not let the model design and improve its own?
A team of researchers from several leading institutions has proposed exactly this idea, introducing a framework where the artifact being evaluated is not the model’s answer, but the **runnable harness the model writes itself**.
## What Is an Agent Harness, and Why Does It Matter?
An agent harness is the complete runtime environment around a language model. It includes the execution loop that governs how the model iterates, the tools it can call (file operations, web search, code execution), the context management system, state tracking mechanisms, recovery logic for handling failures, and verification checks to confirm task completion.
Benchmarks like Terminal-Bench 2.1 reveal that harness choice dramatically affects outcomes. The same model weights can deliver vastly different success rates depending on whether they are wrapped in a simple loop or a sophisticated orchestration framework. Traditional benchmarks keep the harness fixed, meaning they measure the model’s ability to work within a pre-built environment rather than its ability to construct that environment from scratch.
## The Two-Stage Framework: Creation and Evolution
The core innovation is a two-stage process that tests whether a model can go from zero infrastructure to a working, optimized harness.
### Stage One: Creation
In the creation stage, each model receives what amounts to a **minimal starting kit**. The seed environment provides only passive capabilities — reading files, searching, running processes, and writing results and trajectory logs. Crucially, there is no execution loop, no planning mechanism, no verification step, no retry logic, and no stopping rule. As a result, the unmodified seed scores **zero on every benchmark**.
Each creator model is then given a task-family specification, a brief design tutorial, and a small set of development cases (between one and three). From this sparse foundation, the model must build a complete, functional harness from scratch. Once built, the harness is frozen before being evaluated on hidden tasks.
The creation spans four domains — code generation, search and browsing, writing, and machine learning experimentation — across five benchmarks totaling over 2,200 evaluation instances. Each model builds three separate harnesses per benchmark to account for variability.
### Stage Two: Evolution
After creation, the model receives its own frozen harness and is asked to **revise it** using execution feedback. The evolution stage uses a fixed set of 100 software engineering tasks and all 89 terminal tasks to provide signals for improvement. Each official candidate must complete both evaluations as a pair, with a controlled budget of 10 pairs and a maximum of two five-task probes between pairs.
The final test is held-out performance: each version is scored on 630 instances the model has never seen during creation or evolution, providing a true measure of generalization rather than overfitting to the feedback tasks.
Harnesses are evaluated on two axes — **capability** (task success rate) and **efficiency** (executor token consumption, with the model’s own generation tokens excluded to isolate the harness’s operational cost).
## Models and Setup
Six cutting-edge models were tested as harness creators:
– Opus 4.8
– GPT-5.5
– Gemini 3.1 Pro
– DeepSeek V4 Pro
– Qwen 3.7 Max
– Seed 2.0 Pro
All creators operated inside a unified assistant interface, with one model using a slightly different coding environment due to platform constraints.
The evaluation covered 2,207 instances across five benchmarks:
– SWE-bench Pro public split (731 instances)
– Terminal-Bench 2.1 (89 instances)
– MLE-bench (75 instances)
– EQ-Bench3 (46 instances)
– BrowseComp (1,266 instances)
Each harness was tested both under its own creator (Self-Eval) and under a fixed, shared runtime environment using Gemini 3.1 Pro (Unified-Eval). This dual-testing approach reveals how much of a harness’s success depends on the model that built it versus the harness design itself.
## What Did the Models Build?
### The Creation Results
Under self-evaluation, the top performer was Opus 4.8, which achieved an average score of 67.8 — impressive given that its starting point was a zero-scoring seed. The human-engineered reference benchmark score was 86.2, leaving a gap of nearly 18 points.
The performance varied significantly by domain:
**Code tasks** saw strong results, with Opus 4.8 reaching 69.3 on software engineering benchmarks versus the 80.0 reference. In terminal-based tasks, Gemini 3.1 Pro took the lead at 68.8 versus the 88.8 reference.
**Search and browsing** showed the largest gaps. The best browsing score was 52.6, achieved by GPT-5.5, against a reference of 92.2 — a substantial deficit that suggests current models still struggle to build effective search orchestration systems.
**Writing tasks** produced a surprise: Opus 4.8 scored 84.6 on the writing benchmark, actually surpassing the 83.7 reference. This suggests that for structured generation tasks, self-designed harnesses can be competitive with or even exceed hand-built ones.
**Machine learning experimentation** yielded the most dramatic upside: both Opus 4.8 (32.9) and Gemini 3.1 Pro (32.4) beat the 24.0 reference, indicating that automated harness design may unlock capabilities in ML workflows that human-designed systems miss.
Interestingly, **code volume did not predict quality**. Across 18 code harnesses, the models added over 17,000 net lines of code, yet Gemini contributed the fewest (roughly 1,000 lines) and still led its benchmark. Self-test counts showed almost no correlation with scores, while revision call frequency did correlate — up to 0.57 — suggesting that iterative refinement matters more than initial thoroughness.
### The Problem of Dead Code
A revealing finding was how much generated machinery goes unused. Of 108 code component instances across all harnesses, 72 triggered during real runs, but 18 never fired at all. All of the dormant components were related to state and memory management. Eleven of the eighteen harnesses defined a State class, yet across 26,679 recorded trajectories, not a single checkpoint event ever appeared. In the writing domain, 124 of 587 generated features turned out to be dead code — written but never invoked.
This highlights a fundamental challenge: models can create complex systems, but they don’t always build systems they actually need.
## The Executor Swap Experiment
One of the most illuminating parts of the study involved **swapping the execution environment**. The same harness built by one model was then run under a different model’s runtime to isolate the effects of harness design from executor capability.
The results were dramatic. MLE-bench token usage varied by a factor of nearly 19 between models — from 29.3 million tokens to 208.4 million tokens for the same benchmark. GPT-5.5 achieved a 19.1% medal rate with far fewer tokens than DeepSeek V4 Pro’s 19.6% rate.
When executors were swapped, rankings reshuffled significantly. Qwen 3.7 Max gained 17.6 points on browsing and 12.9 points on ML tasks, suggesting that its own executor was the bottleneck rather than its harness design. Conversely, Opus 4.8’s SWE-Pro score plummeted from 69.3 to 33.0 when run under a different executor — partly because one harness had hard-coded a 120-step limit tied to its original runtime. The Opus search harness’s duplicate query rate jumped from 10.1% to 88.2% after the swap, revealing how tightly some harness designs couple to their native execution environment.
## Evolution: Can Harnesses Improve Over Time?
Nine lineages — five using their own runtime for feedback and four using a fixed Gemini executor — produced 73 official versions through 64 adjacent switches between versions.
All five self-runtime creators showed improvement on held-out tasks, with gains ranging from 1.43 to 4.44 points and a mean improvement of 3.11 points. Under the fixed Gemini executor, only Opus showed continued improvement; GPT-5.5 actually regressed by 10.32 points, and several other models stagnated or declined.
Critically, **progress was not monotonic**. Of the 64 version switches:
– 8 regressed on both benchmarks
– 16 regressed on one benchmark
– 27 showed gains that fell within the repeated-run noise band (about ±4.75 points)
– 2 demonstrated clear, unambiguous positive evidence
– 1 involved no executable code change at all
Feedback and held-out scores moved in the same direction only 34 times out of 64 (53.1%) — barely better than a coin flip. Only 2 of the 9 declared final versions were actually optimal on held-out data. Of the 169 new functions or classes introduced during evolution, 25 had no callers whatsoever, meaning they were dead code added through revision.
### The Biggest Win
The clearest success story involved Opus 4.8 noticing that 99 out of 100 runs reported success while only 48 actually passed the hidden test cases. The model traced this discrepancy to premature completion signals and added a completion gate — a verification check that confirmed tasks were genuinely finished before declaring success. This single fix produced meaningful improvement on held-out tasks.
Notably, failure diagnosis was the weakest capability across all models. The dedicated trajectory analysis interface was called only twice across all evolution runs, suggesting that models are far better at adding features than at diagnosing why existing features fail.
## The Interactive Explainer
The research team also built an interactive browser-based tool that walks through the harness development process step by step. It starts with the weak zero-scoring seed, animates the addition of six control modules (execution loop, tool policy, context management, state and memory, lifecycle handling, and verification), and lets users explore creation scores, executor swap effects, and evolution outcomes through visual comparisons. The tool illustrates how dead code accumulates, how scores shift under different executors, and how evolution switches distribute across positive and negative outcomes.
## FAQ
**Q: What exactly is an agent harness?**
A: An agent harness is the complete runtime wrapper around a language model, including the execution loop, available tools, context management, state tracking, failure recovery mechanisms, and verification checks. It determines how the model interacts with the world and processes feedback.
**Q: Why is the HarnessDev approach different from standard benchmarks?**
A: Standard benchmarks keep the harness fixed and measure the model’s outputs within it. HarnessDev makes the harness itself the evaluation target — measuring whether the model can design, build, and iteratively improve the infrastructure that determines how well it (or any model) performs.
**Q: What happens when you start from the “weak seed”?**
A: The weak seed provides only basic capabilities — file reading, search, process execution, and logging — with no loop, planner, verifier, or stopping logic. Unmodified, it scores zero on all benchmarks, serving as a blank canvas for the model to build upon.
**Q: Why does code volume not correlate with harness quality?**
A: The study found that the model producing the most code did not achieve the best results. Some of the most effective harnesses were also the most compact, while larger codebases often contained significant amounts of dead code and unused features.
**Q: What does the executor swap experiment reveal?**
A: Swapping the execution environment while keeping the harness identical shows how much of a harness’s performance depends on its compatibility with the runtime. Some harnesses are tightly coupled to their native executor and perform poorly when moved, while others are more portable.
**Q: Did evolution reliably improve the harnesses?**
A: Self-runtime evolution generally improved held-out performance, but the gains were often small and inconsistent. Feedback signals did not reliably predict held-out outcomes, and only a handful of version switches produced clear, measurable improvements. This suggests that automated harness refinement is still an immature capability.
**Q: What was the weakest part of the models’ harness-building process?**
A: Failure diagnosis was consistently the weakest step. Models were adept at adding new features but struggled to identify why existing features failed. The dedicated diagnostic interface was used almost never, indicating a gap in current models’ ability to introspect on their own systems.
**Q: Can these self-built harnesses outperform human-engineered ones?**
A: In specific domains — particularly writing and ML experimentation — self-built harnesses matched or exceeded human-engineered references. However, in search and browsing tasks, the gap remained large, and overall, human references still set a higher bar across most benchmarks.
## Conclusion
The idea of letting AI models build and optimize their own agent infrastructure represents a significant shift in how we think about evaluation and capability. The results are promising but mixed. Models can create functional harnesses from minimal starting points, occasionally matching or beating human-designed systems in specific domains. However, the process is far from reliable — dead code accumulates, feedback signals are noisy, and the models’ ability to diagnose and fix failures remains weak.
What makes this research particularly valuable is that it exposes the **hidden importance of harness design** in agent performance. Much of what we attribute to a model’s intelligence may actually reflect the quality of the runtime environment it operates within. By making the harness the object of evaluation, this work opens up new questions about how to build systems that are robust, portable, and self-improving.
The gap between current self-built harnesses and human-engineered references remains significant, especially in complex domains like web search and browsing. But the evolutionary results — even if modest and inconsistent — suggest that with better feedback mechanisms and diagnostic tools, AI systems may eventually learn to design their own optimal working environments.
As the field moves toward more autonomous AI agents, understanding and improving the harness becomes not just an engineering concern, but a fundamental research direction.
Thank you for reading
