Your ReAct Agent Is Losing 90% of Its Retries — Right here’s Tips on how to Cease It

Who that is for: ML engineers and AI builders working LLM brokers in manufacturing — particularly ReAct-style methods utilizing LangChain, LangGraph, AutoGen, or customized software loops. For those who’re new to ReAct, it’s a prompting sample the place an LLM alternates between Thought, Motion, and Statement steps to resolve duties utilizing instruments.

are burning nearly all of their retry price range on errors that may by no means succeed.

In a 200-task benchmark, 90.8% of retries had been wasted — not as a result of the mannequin was flawed, however as a result of the system saved retrying instruments that didn’t exist. Not “unlikely to succeed.” Assured to fail.

I didn’t discover this by tuning prompts. I discovered it by instrumenting each retry, classifying each error, and monitoring precisely the place the price range went. The basis trigger turned out to be a single architectural assumption: letting the mannequin select the software identify at runtime.

Right here’s what makes this significantly harmful. Your monitoring dashboard is nearly definitely not displaying it. Proper now it in all probability reveals:

• Success price: fantastic
• Latency: acceptable
• Retries: inside limits

What it doesn’t present: what number of of these retries had been unimaginable from the primary try. That’s the hole this text is about.

Simulation word: All outcomes come from a deterministic simulation utilizing calibrated parameters, not reside API calls. The hallucination price (28%) is a conservative estimate for tool-call hallucination in ReAct-style brokers derived from failure mode evaluation in revealed GPT-4-class benchmarks (Yao et al., 2023; Shinn et al., 2023) — it’s not a instantly reported determine from these papers. Structural conclusions maintain as architectural properties; precise percentages will fluctuate in manufacturing. Full limitations are mentioned on the finish. Reproduce each quantity your self: python app.py --seed 42.

GitHub Repository:

In manufacturing, this implies you’re paying for retries that can’t succeed—and ravenous those that would.

TL;DR

90.8% of retries had been wasted on errors that would by no means succeed. Root trigger: letting the mannequin select software names at runtime (TOOLS.get(tool_name)). Prompts don’t repair it — a hallucinated software identify is a everlasting error. No retry could make a lacking key seem in a dictionary.

Three structural fixes get rid of the issue: classify errors earlier than retrying, use per-tool circuit breakers, transfer software routing into code. Consequence: 0% wasted retries, 3× decrease step variance, predictable execution.

The Legislation This Article Is Constructed On

Earlier than the info, the precept — said as soon as, bluntly:

Retrying solely is sensible for errors that may change. A hallucinated software identify can’t change. Due to this fact, retrying it’s assured waste.

This isn’t a chance argument. It isn’t “hallucinations are rare enough to ignore.” It’s a logical property: TOOLS.get("web_browser") returns None on the primary try, the second, and each try after. The software doesn’t exist. The retry counter doesn’t know that. It burns a price range slot anyway.

The complete downside flows from this mismatch. The repair does too.

The One Line Silently Draining Your Retry Finances

It seems in nearly each ReAct tutorial. You’ve in all probability written it:

tool_fn = TOOLS.get(tool_name)   # ◄─ THE LINE

if tool_fn is None:
    # No error taxonomy right here.
    # TOOL_NOT_FOUND seems to be an identical to a transient community blip.
    # The worldwide retry counter burns price range on a software
    # that may by no means exist — and logs that as a "failure".

That is the road. Every part else on this article follows from it.

When an LLM hallucinates a software identify — web_browser, sql_query, python_repl — TOOLS.get() returns None. The agent is aware of the software doesn’t exist. The worldwide retry counter doesn’t. It treats TOOL_NOT_FOUND identically to TRANSIENT: similar price range slot, similar retry logic, similar backoff.

The cascade: each hallucination consumes retry slots that would have dealt with an actual failure. When a real community timeout arrives two steps later, there may be nothing left. The duty fails — logged as generic retry exhaustion, with no hint of a hallucinated software identify being the foundation trigger.

In case your logs comprise retries on TOOL_NOT_FOUND, you have already got this downside. The one query is what fraction of your price range it’s consuming. On this benchmark, the reply was 90.8%.

The Benchmark Setup

Two brokers, 200 duties, similar simulated parameters, similar instruments, similar failure charges — with one structural distinction.

Comparability word: This benchmark compares a naive ReAct baseline towards a workflow with all three fixes utilized. Fixes 1 (error taxonomy) and a couple of (per-tool circuit breakers) are independently relevant to a ReAct agent with out altering its structure. Repair 3 (deterministic software routing) is the structural differentiator — it’s what makes hallucination on the routing layer unimaginable. The hole proven is cumulative; preserve this in thoughts when studying the numbers.

ReAct agent: Normal Thought → Motion → Statement loop. Single world retry counter (MAX_REACT_RETRIES = 6, MAX_REACT_STEPS = 10). No error taxonomy. Device identify comes from LLM output at runtime. Every hallucinated software identify burns precisely 3 retry slots (HALLUCINATION_RETRY_BURN = 3) — this fixed instantly drives the 90.8% waste determine and is mentioned additional in Limitations.

Managed workflow: Deterministic plan execution the place software routing is a Python dict lookup resolved at plan time. Error taxonomy utilized on the level of failure. Per-tool circuit breakers (journeys after 3 consecutive failures, restoration probe after 5 simulated seconds, closes after 2 probe successes). Retry logic scoped to error class.

Simulation parameters:

This fixed is the direct mechanical driver of the 90.8% waste determine. At a price of 1, fewer slots are burned per occasion — the workflow’s wasted depend stays at 0 regardless. Run the sensitivity test your self: modify this fixed and observe that the workflow all the time wastes zero retries.

The simulation makes use of three instruments — search, calculate, summarise — with lifelike failure charges per software. Device price is tracked at 200 tokens per LLM step.

Each quantity on this article is reproduced precisely by python app.py --seed 42.

What the Benchmark Discovered

Success Charge Hides the Actual Downside

ReAct succeeded on 179/200 duties (89.5%). The workflow succeeded on 200/200 (100.0%).

The ten.5% hole is actual. However success price is a go/fail metric — it says nothing about how near the sting a passing run got here, or what it burned to get there. The extra informative quantity is what occurred inside these 179 “successful” ReAct runs. Particularly: the place did the retry price range go?

The Retry Finances

466 of 513 retries — 90.8% — focused errors that can’t succeed by definition. The workflow fired 80 retries. Each single one was helpful. The hole is 6.4× in whole retries and 466-to-0 in wasted ones. That isn’t a efficiency distinction. It’s a structural one.

A word on the mechanics: HALLUCINATION_RETRY_BURN = 3 means every hallucinated software identify burns precisely 3 retry slots within the ReAct simulation. The 90.8% determine is delicate to this fixed — at a price of 1, fewer retries are wasted per hallucination occasion. However the structural property holds at each worth: the workflow wastes zero retries regardless, as a result of non-retryable errors are categorized and skipped earlier than any slot is consumed. Run the sensitivity test your self: modify HALLUCINATION_RETRY_BURN and observe that the workflow’s wasted depend stays at 0.

Why 19 of 21 ReAct Failures Had Similar Root Causes

19 of 21 failures: hallucinated software identify, world retry price range exhausted, activity lifeless. Not community failures. Not price limits. Hallucinated strings retried till nothing was left. The workflow had zero failures throughout 200 duties.

Your success price dashboard won’t ever floor this. The failure purpose is buried contained in the retry loop with no taxonomy to extract it. That’s the dashboard blindness the title guarantees — and it’s worse than it sounds, as a result of it means you haven’t any sign when issues are degrading, solely once they’ve already failed.

The Error Taxonomy: From “Unknown” to Absolutely Labeled

The basis repair is classifying errors on the level they’re raised. Three classes are retryable; three are usually not:

# Retryable — can succeed on a subsequent try
RETRYABLE = {TRANSIENT, RATE_LIMITED, DEPENDENCY_DOWN}

# Non-retryable — retrying wastes price range by definition
NON_RETRYABLE = {INVALID_INPUT, TOOL_NOT_FOUND, BUDGET_EXCEEDED}

When each error carries a category, the retry determination turns into one line:

if not exc.is_retryable():
    log(RETRY_SKIPPED)   # zero price range consumed
    break

The complete taxonomy from the 200-task run:

ReAct’s dominant occasion is hallucination — 155 occasions, all non-retryable, all burning price range. The workflow’s dominant occasion is circuit_open — 49 fast-fails that by no means touched an upstream service. The workflow logged zero hallucination occasions as a result of it by no means asks the mannequin to provide a software identify string.

You can not hallucinate a key in a dict you by no means ask the mannequin to provide.

That is an architectural assure inside the simulation design. In an actual system the place the LLM contributes to plan technology, hallucinations might nonetheless happen upstream of software routing. The assure holds exactly the place routing is totally deterministic and the mannequin’s output is proscribed to plan construction — not software identify strings.

The eight loop_detected occasions in ReAct come from a 18% loop price utilized when len(historical past) > 2 — the mannequin “decides to think more” fairly than act, consuming a step with out calling a software. The workflow has no equal as a result of it doesn’t give the mannequin step-selection authority.

Step predictability: the hidden instability σ reveals

The means are practically an identical. The distributions are usually not. Normal deviation is 3× increased for ReAct.

Workflow σ holds at 0.46 throughout all hallucination charges examined — not by coincidence, however as a result of plan construction is fastened. Process sort (math, abstract, search) determines step depend at plan time. The hallucination roll doesn’t have an effect on step depend when software routing by no means passes via the mannequin’s output.

In manufacturing, excessive σ means: unpredictable latency (SLAs can’t be dedicated to), unpredictable token price (price range forecasts are inaccurate), and invisible burst load (a foul cluster of long-running duties arrives with no warning). Predictability is a manufacturing property. Success price doesn’t measure it. σ does.

The Three Structural Fixes

Repair 1: Classify Errors Earlier than Deciding Whether or not to Retry

The basis repair is classifying errors on the level they’re raised. Three classes are retryable; three are usually not:

def call_tool_with_retry(tool_name, args, logger, ledger,
                         step, max_retries=2, fallback=None):
    for try in vary(max_retries + 1):
        attempt:
            return call_tool_with_circuit_breaker(tool_name, args, ...)
        besides AgentError as exc:
            if not exc.is_retryable():
                # Non-retryable: RETRY_SKIPPED — zero price range consumed
                logger.log(RETRY_SKIPPED, error_kind=exc.sort.worth)
                break                          # ← this line drops waste to 0
            if try < max_retries:
                ledger.add_retry(wasted=False)
                backoff = min(0.1 * (2 ** try) + jitter, 2.0)
                logger.log(RETRY, try=try, backoff=backoff)
    if fallback:
        return ToolResult(tool_name, fallback, 0.0, is_fallback=True)
    elevate last_error

RETRY_SKIPPED is the audit occasion that proves taxonomy is working. Search your manufacturing logs for it to see precisely which non-retryable errors had been caught at which step, by which activity, with zero price range consumed. ReAct can’t emit this occasion — it has no taxonomy to skip from.

This repair is relevant to a ReAct agent immediately with out altering its software routing structure. For those who run LangChain or AutoGen, you possibly can add error classification to your software layer and scope your retry decorator to TransientToolError with out touching the rest. It won’t get rid of hallucination-driven waste totally — that requires Repair 3 — nevertheless it prevents INVALID_INPUT and different everlasting errors from burning retries on makes an attempt that additionally can’t succeed.

Repair 2: Per-Device Circuit Breakers As a substitute of a International Counter

A worldwide retry counter treats all instruments as a single failure area. When one software degrades, it drains the price range for each different software. Per-tool circuit breakers comprise failure regionally:

# Every software will get its personal circuit breaker occasion
# CLOSED    → calls go via usually
# OPEN      → calls fail instantly, no upstream hit, no price range consumed
# HALF-OPEN → one probe name; if it succeeds, circuit closes

class CircuitBreaker:
    failure_threshold: int   = 3    # journeys after 3 consecutive failures
    recovery_timeout:  float = 5.0  # simulated seconds earlier than probe allowed
    success_threshold: int   = 2    # probe successes wanted to shut

The benchmark logged 49 CIRCUIT_OPEN occasions for the workflow — each one a name that fast-failed with out touching a degraded upstream service and with out consuming retry price range. ReAct logged zero, as a result of it has no per-tool state. It hammers a degraded software till the worldwide price range is gone.

Like Repair 1, that is independently relevant to a ReAct agent. Per-tool circuit breakers wrap the software name layer no matter how the software was chosen. Threshold values will want tuning to your workload.

Repair 3: Deterministic Device Routing (The Structural Differentiator)

That is the repair that eliminates the hallucination downside on the routing layer. Fixes 1 and a couple of cut back the injury from hallucinations; Repair 3 makes them structurally unimaginable the place it’s utilized.

# ReAct — software identify comes from LLM output, could be any string
tool_name = llm_response.tool_name       # "web_browser", "sql_query", ...
tool_fn   = TOOLS.get(tool_name)         # None if hallucinated → price range burns

# Workflow — software identify resolved from plan at activity begin, all the time legitimate
STEP_TO_TOOL = {
    StepKind.SEARCH:    "search",
    StepKind.CALCULATE: "calculate",
    StepKind.SUMMARISE: "summarise",
}
tool_name = STEP_TO_TOOL[step.kind]      # KeyError is unimaginable; hallucination is unimaginable

Use the LLM for reasoning — what steps are wanted, in what order, with what arguments. Use Python for software routing. The mannequin contributes plan construction (step varieties), not software identify strings.

The trade-off is value naming actually: deterministic routing requires that your activity construction maps onto a finite set of step varieties. For open-ended brokers that must dynamically compose novel software sequences throughout a big registry, this constrains flexibility. For methods with predictable activity buildings — nearly all of manufacturing deployments — the reliability and predictability positive factors are substantial.

Earlier than/after abstract:

The Sensitivity Evaluation: The 5% Consequence Is the Alarming One

The 5% row deserves explicit consideration. ReAct reveals 100% success — your monitoring reviews a wholesome agent. However 54.7% of retries are nonetheless wasted. The price range is quietly draining.

That is the dashboard blindness made exact. When an actual failure cluster arrives — a price restrict spike, a degraded service, a quick outage — lower than half your designed retry capability is out there to deal with it. You’ll not see this coming. Your success price was 100% till the second it wasn’t.

The workflow wastes 0% of retries at each price examined. The σ holds at 0.46 no matter hallucination frequency. These are usually not rate-dependent enhancements — they’re properties of the structure.

Latency: What the CDF Reveals That Averages Conceal

The workflow seems slower on common as a result of failed ReAct runs exit early — they appear quick as a result of they failed quick, not as a result of they accomplished effectively. At P95 — the metric that issues for SLA commitments — the latency is successfully an identical: 143.3ms versus 146.2ms.

You aren’t buying and selling tail latency for reliability. On the tail, the simulation reveals you possibly can have each. Token price favors the workflow by 6.6%, as a result of it doesn’t burn LLM steps on hallucination-retry loops that produce no helpful output.

Three Diagnostic Questions for Your System Proper Now

Earlier than studying the implementation steerage, reply these three questions on your present agent:

1. When a software identify from the mannequin doesn’t match any registered software, does your system retry? If sure, price range is draining on non-retryable errors proper now.

2. Is your retry counter world or per-tool? A worldwide counter lets one degraded software exhaust the price range for all others.

3. Are you able to search your logs for RETRY_SKIPPED or an equal occasion? If not, your system has no error taxonomy and no audit path for wasted price range.

For those who answered “yes / global / no” to those three — Repair 1 and Repair 2 are the quickest path to restoration, relevant with out altering your agent structure.

Implementing This in Your Stack At present

These three fixes could be utilized incrementally to any framework — LangChain, LangGraph, AutoGen, or a customized software loop.

Step 1 — Add error classification (half-hour). Outline two exception lessons in your software layer: one for retryable errors (TransientToolError), one for everlasting ones (ToolNotFoundError, InvalidInputError). Elevate the suitable class on the level the error is detected.

Step 2 — Scope retries to error class (quarter-hour). For those who use tenacity, swap retry_if_exception for retry_if_exception_type(TransientToolError). For those who use a customized loop, add if not exc.is_retryable(): break earlier than the retry increment.

Step 3 — Transfer software routing right into a dict (1 hour). When you have a hard and fast activity construction, outline it as a StepKind enum and resolve software names from dict[StepKind, str] at plan time. Non-obligatory in case your use case requires open-ended software composition, nevertheless it eliminates hallucination-driven price range waste totally the place it may be utilized.

Here’s what the vulnerability seems to be like in LangChain, and the best way to repair it:

Susceptible sample:

from langchain.brokers import AgentExecutor, create_react_agent

# If the mannequin outputs "web_search" as a substitute of "search",
# AgentExecutor will retry the step earlier than failing —
# consuming price range on an error that can't succeed.
executor = AgentExecutor(
    agent=create_react_agent(llm, instruments, immediate),
    instruments=instruments,
    max_iterations=10
)
executor.invoke({"input": activity})

Mounted sample — error taxonomy + deterministic routing:

from tenacity import retry, stop_after_attempt, retry_if_exception_type

class ToolNotFoundError(Exception): go   # non-retryable
class TransientToolError(Exception): go  # retryable

# Device routing in Python — mannequin outputs step sort, not software identify
TOOL_REGISTRY = {"search": search_fn, "calculate": calc_fn}

def call_tool(identify: str, args: str):
    fn = TOOL_REGISTRY.get(identify)
    if fn is None:
        elevate ToolNotFoundError(f"'{name}' not registered")  # by no means retried
    attempt:
        return fn(args)
    besides RateLimitError as e:
        elevate TransientToolError(str(e))   # retried with backoff

@retry(
    cease=stop_after_attempt(3),
    retry=retry_if_exception_type(TransientToolError)
)
def run_step(tool_name: str, args: str):
    return call_tool(tool_name, args)

Manufacturing word: The eval() name within the benchmark’s tool_calculate is current for simulation functions solely. By no means use eval() in a manufacturing software — it’s a code injection vulnerability. Change it with a protected expression parser similar to simpleeval or a purpose-built math library.

Benchmark Limitations

Hallucination price is a parameter, not a measurement. The 28% determine is a conservative estimate derived from failure mode evaluation in Yao et al. (2023) and Shinn et al. (2023) — not a instantly reported determine from both paper. A well-prompted mannequin with a clear software schema and a small, well-named software registry could hallucinate software names far much less ceaselessly. Run the benchmark at your precise noticed price.

HALLUCINATION_RETRY_BURN is a simulation fixed that drives the waste proportion. At a price of 1, fewer retries are wasted per hallucination occasion; the 90.8% determine can be decrease. The structural conclusion — the workflow wastes 0% in any respect values — holds regardless. Run python app.py --seed 42 with modified values of 1 and a couple of to confirm.

The workflow’s zero hallucination depend is a simulation design property. Device routing by no means passes via LLM output on this benchmark. In an actual system the place the LLM contributes to plan technology, hallucinations might happen upstream of routing.

Three instruments is a simplified surroundings. Manufacturing brokers sometimes handle dozens of instruments with heterogeneous failure modes. The taxonomy and circuit breaker patterns scale properly; threshold values will want tuning to your workload.

Latency figures are simulated. The P95 near-equivalence is the production-relevant discovering. Absolute millisecond values mustn’t inform capability planning. Common latency comparisons are confounded by early-exit failures in ReAct and per-step LLM accounting within the workflow — use P95 for any latency reasoning.

Full Metrics

Full per-metric outcomes for all 200 duties (seed=42, hallucination_rate=28%) can be found in `experiment_results.json` within the GitHub repository. Run `python app.py -seed 42 -export-json` to regenerate them regionally.

References

• Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, Okay., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Appearing in Language Fashions. ICLR 2023.
• Shinn, N., Cassano, F., Gopinath, A., Narasimhan, Okay., & Yao, S. (2023). Reflexion: Language Brokers with Verbal Reinforcement Studying. NeurIPS 2023.
• Fowler, M. (2014). CircuitBreaker. martinfowler.com.
• Nygard, M. T. (2018). Launch It! Design and Deploy Manufacturing-Prepared Software program (2nd ed.). Pragmatic Bookshelf.
• Sculley, D., et al. (2015). Hidden technical debt in machine studying methods. NeurIPS 2015.

Disclosure

Simulation methodology. All outcomes are produced by a deterministic simulation (python app.py --seed 42), not reside API calls. The 28% hallucination price is a calibrated parameter derived from failure mode evaluation in revealed benchmarks — not a instantly measured determine from reside mannequin outputs.

No conflicts of curiosity. The writer has no monetary relationship with any software, framework, mannequin supplier, or firm talked about on this article. No merchandise are endorsed or sponsored.

Unique work. This text, its benchmark design, and its code are the writer’s authentic work. References are used solely to attribute revealed findings that knowledgeable calibration and design.

GitHub:

python app.py --seed 42 — full outcomes and all six figures. python app.py --replay 7 — verbose single-task execution, step-by-step.