**The Real Test for Agent Survival: Cost Per Successful Outcome**
In the world of AI agents, the twelve-metric evaluation harness has long been the gold standard for determining whether an agent is “working.” For months, teams celebrated green dashboards showing strong task completion, high faithfulness, and low hallucination rates. But a stark reality check came when a mid-market SaaS company built an agent to handle billing and account tickets.
Their agent performed well on paper. Every metric the harness tracked was within target. Yet when the CFO reviewed the numbers, the picture changed: the cost per resolved ticket was higher than what the company paid humans to do the same work. The agent was accurate—but not economical. It was shut down.
This is the gap the original harness missed. It measured whether the agent was right, but not whether the agent paid for itself. In 2026, that second question decides which agents survive—and it’s not in the framework.
—
### What the Harness Measures, and the Half It Misses
The twelve-metric harness was designed to answer one question: Is this agent behaving correctly in production? It covers critical failure modes—wrong retrieval, hallucinated generation, bad tool calls, reasoning loops, and drift from user intent. These are quality metrics, and they matter.
But none of them are economic.
When the framework was created, cost felt like an afterthought. Inference was expected to get cheaper, while reliability was the hard problem. But agent architectures evolved differently. Modern agents chain multiple reasoning passes, tool calls, retries, and self-critiques—each adding tokens and time. Inference per token fell, but tokens per resolution rose, and for many workflows, costs climbed.
The result? An agent can sit inside every quality band and still lose money on every unit of work. The harness shows all green while the unit economics bleed.
—
### The Metric: Cost Per Successful Outcome
The harness needed one more metric: **cost per successful outcome**—the fully-loaded cost of producing one successful business result, divided by the number of successful outcomes.
The key word is *successful*. Cost per call is easy but misleading. Cost per successful outcome is harder but honest.
Here’s how the math played out for the SaaS company:
– Cost per attempt: ~$3.40
– Resolution rate: 71%
– Escalation rate: 29%
– Human escalation cost: ~$4.20 per ticket
Across 1,000 tickets:
– Agent spend: $3,400
– Escalation spend: $1,218
– Total: $4,618
– Human baseline: $4,200
Cost per successful agent outcome:
– $3,400 ÷ 710 resolved = **$4.79**
– Human cost: $4.20
The agent was more accurate—but also more expensive per win. The harness reported success. The CFO saw failure. They were both right, because they measured different things.
—
### Why It’s Harder to Measure Than It Looks
Three issues make this metric difficult:
1. **Attribution of failed attempts**
The cost of failures contributes to the cost of success. If an agent tries and fails, then escalates, that $3.40 is part of the cost of the eventual success. Most dashboards ignore this.
2. **Double payment on escalation**
Failed agent attempts don’t replace human work—they add to it. Below a certain resolution rate, agents become a tax rather than a substitute.
3. **Defining the value of an outcome**
Cost per success only matters when compared to what that success is worth. For support tickets, that’s the human cost avoided. For sales or security workflows, it’s a modeling exercise—but it’s never irrelevant.
These aren’t excuses to skip the metric. They’re reasons why it gets skipped—and why accurate agents get shut down.
—
### The Threshold That Decides Survival
Once you can compute cost per successful outcome, the test is simple: it must sit below the business value of that outcome, with room for operational overhead.
For the SaaS company:
– Value of a resolved ticket: ~$4.20
– Cost per successful outcome: $4.79
– Result: underwater
This threshold moves against you as scale increases. Traditional software gets cheaper per unit. Agent workflows often do not. Pilot-scale “close enough” can become six-figure losses at production scale.
Teams that keep agents running don’t have the most sophisticated models—they have workflows where successful outcomes are worth more than the cost to produce them.
—
### What Teams Do When the Number Comes Back Underwater
An underwater result isn’t automatic shutdown—it’s a signal to change something.
1. **Reduce cost per attempt**
Trim reasoning depth, cache repeated retrievals, reduce retries. One team cut cost per attempt from $3.40 to $2.30 with no quality impact.
2. **Raise resolution rate**
This is slow but compounds—better prompts, better routing, better guardrails.
3. **Narrow deployment scope**
Deploy only on ticket types where cost per success is already below the value line. Route the rest to humans.
But beware: routing hard work to cheaper models can reintroduce quality failures. Cost and quality are coupled. Optimize both together.
In the SaaS case, the team:
– Cut attempt cost to $2.30
– Narrowed scope to billing tickets
– Routed complex disputes to humans
Result: cost per successful outcome fell to $3.10—below the $4.20 value line. The agent survived.
—
### The Same Metric, Pointed the Other Way
Cost per successful outcome isn’t just a kill switch—it’s a truth detector.
A legal team built an agent to flag risk clauses in contracts. Cost per attempt: $18. On a cost-per-call dashboard, that flagged for throttling. On a cost-per-success dashboard, it told a different story.
The agent caught one genuine risk per nine contracts. A missed clause could mean five- or six-figure exposure. Human review had been letting some through. At $162 per caught clause against tens of thousands of dollars of risk exposure, the $18 run looked cheap.
The team widened the scope. The expensive per-call number became a badge of value.
—
### Instrumenting It Without a Rebuild
You don’t need a new dashboard. You need three things joined by a trace ID:
– Per-attempt cost (from billing or instrumentation)
– Outcome label (success, escalation, failure)
– Stable attempt identifier (to sum retries into one attempt)
The value side needs one defensible number: what a successful outcome is worth. For human-displacement workflows, use the loaded cost of the human alternative. For risk avoidance, use historical loss data. It doesn’t need to be precise—just defensible.
Once in place, cost per successful outcome is a single query:
– Total agent spend ÷ successful outcomes vs. value line
—
### The Metric I Should Have Included
The twelve-metric harness answers whether an agent is behaving correctly. But production teams face a second question: Is a successful outcome worth the cost?
In 2026, both answers must be yes for an agent to survive. Accuracy tells you the agent can do the work. Cost per successful outcome tells you whether the work, done this way, is worth doing.
If I published the harness today, cost per successful outcome would be the thirteenth metric—equal in importance to task completion, because together they decide whether an agent is a capability or a liability.
The fastest way to find out which agents those are isn’t to wait for the CFO’s review. It’s to measure the number first.
—
**FAQ**
**Q: Can’t I just lower agent costs by switching to a cheaper model?**
A: You can, but if the cheaper model reduces resolution rate or increases failures, you may simply shift cost from inference to escalations. Only cost per successful outcome tells the full story.
**Q: How do I value an outcome that doesn’t directly replace human work?**
A: Use historical data—how often has a missed risk caused losses? How much time does a human analyst spend on equivalent decisions? Value is a model, but an informed one is better than no number at all.
**Q: Do I need to rebuild my evaluation harness to add this metric?**
A: No. Most inputs are already logged. You need a trace ID, cost data, outcome labels, and a value estimate—not a new system.
**Q: Is cost per successful outcome useful at pilot stage?**
A: Yes. If the number is above value at pilot scale, scaling will only magnify the loss. Catch it early.
**Q: Does this mean accuracy no longer matters?**
A: No. Accuracy is necessary—but not sufficient. An agent must be both right and economical.
—
**Conclusion**
The twelve-metric evaluation harness exposed a critical blind spot in AI agent deployment: it measured correctness but not economics. In 2026, cost per successful outcome is the metric that decides survival. It reveals whether an accurate agent is also a good investment—and it highlights workflows where quality and cost must be optimized together, not in isolation. The agents that last will be those measured not just on what they get right, but on what their success actually costs.
