## Understanding Cost Explosions in Multi-Agent Systems: Beyond Routing and Retries
The rise of agentic architectures—systems where specialized agents collaborate through frameworks like LangGraph—promises better modularity and clearer separation of concerns. However, our experience with a LangGraph-based supervisor-and-worker pattern revealed a challenging truth: introducing orchestration can dramatically increase costs in ways that aren’t immediately obvious. What looked like a clean architectural win on paper masked a subtle but significant spike in token usage, highlighting a critical gap in how we were evaluating and designing these systems.
The increase wasn’t marginal; it was approximately a threefold jump for tasks that hadn’t fundamentally changed in complexity. This counterintuitive result underscores a key insight: **the cost of orchestration isn’t just about making more calls—it’s about multiplying machinery that operates before, around, and after the actual task.** The supervisor agent, multiple system prompts, duplicated context, and the inherent overhead of multi-step coordination all contribute to an “invisible” cost layer that can easily be overlooked when the focus is purely on functional decomposition.
### When Intuition Fails and Assumptions Lead Us Astray
Our initial instinct pointed toward fan-out: the supervisor was likely spawning too many sub-agent calls. Investing time in analyzing call patterns felt like the logical step, but the data showed the branching factor was roughly as designed. This dead end is a classic example of how a seemingly obvious hypothesis can consume valuable time without addressing the core issue.
The true culprit emerged only through deep log analysis: **silent cascading retries**. A validation failure in one agent didn’t just cause a single retry. It forced the *re-execution of its entire upstream context* to reconstruct inputs, even when the final output was ultimately correct. A retry decorator, which seems harmless on an isolated call, becomes a cost multiplier inside a graph with shared state. The retry logic was doing exactly what it was told—retrying on failure—but lacked any awareness of the token economics of that failure. This illustrates a crucial rule: **retries in multi-agent systems must account for graph-aware state rebuilding, not just idempotent re-invocation.**
### Beyond Band-Aids: Why Simple Fixes Fall Short
A common recommendation for retry-related cost inflation is capping attempts and adding exponential backoff. We implemented exactly that. While prudent, these measures only addressed *frequency* of retries, not the *cost per retry*. The most powerful model in the graph was still invoked for every retry, regardless of whether the failure was a trivial schema mismatch or a genuine reasoning hurdle. The mistake was treating all retries as equally costly, leading to a premium bill for problems that didn’t require premium-grade remediation.
### The Real Solution: A Layered Strategy
The breakthrough came from implementing a combination of three interdependent changes:
1. **Routing by Task Complexity:** Instead of a one-model-fits-all approach, we matched the agent’s model to the actual task. Mechanical steps—summarization, formatting, tool call generation—were offloaded to a cheaper, faster model. Only complex, multi-step reasoning or high-stakes judgments retained the most capable (and expensive) model. This required building a routing table, a small maintenance cost that paid significant dividends in efficiency. It forced a conscious decision about *why* a model was needed for a step, rather than defaulting to the best available.
2. **Context Trimming Between Handoffs:** We drastically reduced the context passed forward. Each agent now receives only the specific fields it needs—final results or specific output schema fields—not the entire upstream reasoning trace. The implementation was simple, but the impact was systemic: every call in the graph stopped paying input-token costs for irrelevant historical data. This is a powerful pattern for reducing hidden overhead in agent graphs.
3. **Parallel Execution of Independent Branches:** We parallelized agent calls that had no dependency chain, primarily to reduce latency. The unforeseen benefit was a significant reduction in retry blast radius. In a sequential graph, a failure in a downstream agent could invalidate and re-run entire upstream chains. Parallel, independent branches contain failures, preventing expensive re-computation of unrelated, already-valid work.
### The Measured Impact
The results validated the approach, though with important caveats. Due to a lack of prior per-step cost tracking, the baseline was approximate. Nevertheless, we observed a **roughly 40% reduction in token usage per model** and a **45–55% decrease in end-to-end latency**. The most significant takeaway was that these improvements were deeply interconnected; isolating any single change would have yielded only marginal gains.
### Key Lessons and Forward Considerations
The biggest lesson isn’t about choosing smaller models or capping retries. It’s this: **in multi-agent systems, failures and inefficiencies propagate through graph-aware ways that single-agent pipelines do not.** Every retry, context handoff, and routing decision becomes a variable in your cost model. If you measure only latency and correctness, you will discover these costs in production, often at the worst time.
Looking ahead, the next evolution may be an automatically learned, complexity-based routing classifier. However, this introduces its own cost—a model call to make a routing decision. Whether that trade-off is worthwhile depends on the variability of your task mix. For now, a manually maintained, thoughtful routing table, combined with graph-aware retry logic and context trimming, provides a robust and effective foundation. The era of agentic efficiency requires us to design not just for correctness, but for the true, systemic cost of coordination.
