Below is a newly written article built from the provided content, followed by a concise FAQ and a concluding section.
—
## Article: How the Harness, Not Just the Model, Determines Agent Quality and Cost
Most teams assume that model choice is the primary lever for agent performance. In LangChain’s Terminal-Bench experiment, however, changing only the harness—while keeping the same model throughout—moved a coding agent from roughly 30th place into the top five. This reframes the problem: if the harness determines quality, then *how you run the loop* is an architecture decision, not a deployment detail.
Paul Iusztin’s open-source course “Building a Coding Agent From Scratch,” published through Decoding AI, introduces a small Python agent named **Decode**. The course separates the agent into three run modes, each with different latency profiles and infrastructure needs. Understanding these modes is key to optimizing both performance and cost.
### One Headless Core, Three Shapes
At the center is a **headless harness**—no UI of its own—running the universal agent loop: the LLM picks an action, a tool executes, and the observation feeds back into the context window. Everything else—memory, skills, sandbox, permissions, LSP feedback, and compaction—is harness logic.
Interfaces plug into this core, giving rise to three distinct modes:
#### Mode 1: Interactive, Online
A terminal UI is wired to a live, in-memory session with streaming token-by-token output. The key challenge is steering: injecting input mid-turn can corrupt execution. Decode addresses this with a **steering queue and priority gate**, allowing safe intervention at defined boundaries.
– **Enter** steers immediately within the turn.
– **Alt+Enter** queues a follow-up until the turn ends.
– **Esc** triggers a cooperative abort at the next safe point.
Because a human reads every token, this mode is latency-sensitive and belongs on low-latency hosted APIs.
#### Mode 2: Remote, Offline
Here, the headless harness runs on a server through an agent runtime—Decode uses Kitaru on Modal—deployed to GCP with agents executing on Modal. No one is watching; a backlog of tickets fans out to multiple harnesses in parallel. Each run records progress step by step, enabling resumption after failures without restarting.
The focus is throughput per dollar. Tools execute inside remote Docker sandboxes, and the primary metric is cost efficiency, not time-to-first-token. Pausing for human input freezes the run without burning compute.
#### Mode 3: Async, Online
This mode sits between the two. A live session hands work to a job queue and returns immediately. Background workers fan out to perform LLM calls and post results back later. The user is online but not watching each step. This pattern suits Slack-triggered agents and background PR review, billing like batch rather than chat.
### The Interactive Explainer
An embedded explainer visualizes the three modes, showing how work moves through steering queues, headless harnesses, and job queues. The same harness runs differently based on interface choices—emphasizing that behavior, not just model weights, drives outcomes.
### Why the Provider Changes with the Mode
Cost follows latency and throughput requirements.
– **Interactive work** is latency-bound, so per-token billing makes sense because a human is waiting.
– **Offline and async work** are throughput-bound, so GPU-hour billing is more economical at scale.
For example, processing 1,000 documents of 30,000 input tokens and 500 output tokens costs about **$97** on frontier APIs but roughly **$13** when run on batched serverless GPUs. Prompt caching cannot fully rescue API costs here because each document is a unique prefix. Conversely, leaving an interactive agent idle overnight can cost **$45** in GPU-hour waste, whereas per-token billing would remain low.
A second axis matters too: **serverless vs reserved capacity**. Serverless pricing follows the demand curve, while reservations charge the peak rate for the entire contract. When peak-to-average demand exceeds the reservation discount—which often ranges from 2–5×—serverless is cheaper. Industry data suggests reservation utilization frequently falls below 30%, sometimes under 10%, favoring serverless for bursty, agentic workloads.
### Key Takeaways
– Harness engineering matters more than model tweaks; changing the harness alone moved agent rankings dramatically.
– Interactive modes are latency-bound and suited to per-token pricing.
– Offline and async modes are throughput-bound and more cost-efficient on GPU-hour billing.
– For high-volume batch processing, serverless GPU execution can significantly outperform API pricing.
– Serverless wins when demand variability is high; reservations make sense for steady, predictable loads.
### FAQ
**Q: What is a harness in agent engineering?**
A harness is the runtime loop that controls how an agent selects actions, executes tools, and feeds observations back to the model. It manages context, tool calling, and turn boundaries independently of the model itself.
**Q: Why did changing the harness improve Terminal-Bench scores more than changing the model?**
Because the harness determines how effectively the model can use tools, handle errors, and steer execution. A better orchestration loop reduces wasted steps and improves task completion rates.
**Q: What are the three run modes in Decode?**
The three modes are:
1. Interactive, online (latency-sensitive, live terminal)
2. Remote, offline (throughput-focused, no human watching)
3. Async, online (job-queued, results posted back later)
**Q: Why is per-token billing used for interactive agents?**
Humans read each token as it arrives, so latency matters. You pay for what you consume in time, not for idle capacity.
**Q: Why is GPU-hour billing better for batch or async workloads?**
Because throughput is the goal and idle time must be amortized. Batching work and keeping GPUs busy lowers the effective cost per task.
**Q: When is serverless GPU cheaper than reserved capacity?**
Serverless is cheaper when your peak-to-average demand ratio exceeds the reservation discount—often when demand is bursty or unpredictable.
**Q: Can prompt caching solve high API costs?**
It helps when prompts repeat across requests, but less so when every input is unique, as with many document-level agent tasks.
### Conclusion
The performance and cost of agent workflows depend less on model choice and more on how the agent loop is orchestrated. By matching the harness and execution mode to your latency, throughput, and billing requirements, teams can achieve better results at lower cost—proving that in agent engineering, *how you run the loop is just as important as which model you run*.
—
