# Building Reliable Customer-Facing LLM Workflows: Lessons from Processing Billions of Tokens at Scale
Inside your own company, almost any failure is cheap: you retry, fall back, or potentially even ignore it. Put that same workflow behind a customer’s API or MCP server and the grace is gone. Now only one thing matters: did the customer get a correct, usable result? Their process depends on yours delivering one. They, not you, now decide what counts as delivered. At Databook we process billions of tokens for the world’s largest enterprises; this article is based on real data from production flows at scale. I hope it offers you some useful insights.
## The Four Ways an LLM Call Fails
Delivering that result is harder than it looks, because LLMs are notoriously unreliable. They fail frequently, in four flavors: an invalid answer (empty, unparseable, or simply wrong), a hard error, no answer at all, or no answer in time. And the whole run only succeeds if *every* step does, so the more you chain together, the more chances there are for one of them to fail. A workflow of individually excellent steps can still come out a coin flip.
**FIGURE 1 – The four ways an LLM call fails.** Three are loud — an invalid answer, a hard error, no answer at all — and you see and handle each. The fourth is quiet: a correct answer that simply arrives too late, which looks like success on your side and like failure on the customer’s.
## The Three Budgets You Don’t Control
Inside your own company you can absorb every one of these, because you have slack on every axis: retry the failed step, wait out the slow one, spend a little more, relax the bar if you must. Put the same workflow behind a customer’s API and the slack vanishes, because the run now has to clear three resource budgets *at the same time*, none of which you set:
– **Time** — a window that closes whether or not you’re done: a hard gateway timeout (one to three minutes, sometimes five) that severs the connection mid-run, or something softer: an SLA, a caller blocked on the result, a process that can only wait so long. And it doesn’t resume: when the window closes, the customer just retries, starting the whole run over from zero.
– **Cost** — now a *margin*, not a pool. Every run carries a price the customer already paid, so it has to come back *profitable*, not merely affordable. And the customer, not you, decides how often it runs.
– **Tokens and rate** — a per-minute token budget (TPM) you share across every customer at once, and they tend to call in the same bursts. You hit the ceiling exactly when load is heaviest, which is exactly when latency is worst.
Under all three sits a hard floor you never trade beneath: **quality**. The answer has to be right to count at all. A fast, cheap, on-time answer that’s *wrong* is still a failure. Quality isn’t a budget you spend down.
**FIGURE 2 – The three resource budgets a customer-facing run spends *simultaneously*** — **time**, **cost**, and **token/rate** — resting on a fixed **quality** floor. Each budget is imposed from outside; the floor is the one line no trade may cross.
## The Tension Between Budgets
Any one of these you could manage on its own. The bind is that they apply together and pull against each other, so the obvious fix for one spends another. Wait out a slow step and you blow the time window. Race a second copy to beat the clock and you burn cost and quota. Reach for a stronger model to clear the quality floor and you get slower. None of the budgets are yours to loosen, so the only move left is to trade deliberately across all of them at once — without ever dropping below the floor.
That is what makes a customer-facing workflow a genuinely different thing to build, and it sometimes forces a playbook that, from the inside, looks totally backwards:
– Kill a call that hasn’t failed
– Fire a duplicate of a call you’re already paying for
– Drop to a weaker model on purpose
Within your own walls you’d never bother. You’d just let the slow step finish. And the budget that punishes you most quietly is time: miss it and nothing looks broken on your side. A perfect answer that lands a few seconds late still reads as a success on your dashboards and as a failure to the customer, and it’s the one limit nothing in the stack enforces for you.
## The Core Thesis: Variance Over Speed
Here’s the thesis, up front, because everything else serves it: **once quality clears the bar, reliable delivery is a question of variance, not speed.** A predictable completion time beats a fast one with a long tail, because your customers can’t run their infrastructure on your best case; they have to build for your worst.
## What This Is — and Isn’t: Workflows, Not Free Reasoning Agents
One distinction up front, because it changes everything. This is about an **agentic workflow**: a known process flow with LLM-powered steps inside it, run by a deterministic orchestrator. It is *not* a **reasoning agent** that decides its own next move at runtime. For the same task, a workflow is simply faster: it already knows the plan, skips the deliberation, and runs every independent step in parallel, so it reaches the same answer in a fraction of the time and cost a reasoning agent would take. Both have their place (reasoning agents are much more flexible), but they fail differently and you fix them differently. A reasoning agent’s problem is deciding what to do; a workflow’s problem (the one customers feel) is delivering what it already knows how to do, with quality, and in time. This article is about the latter.
## How Our System Is Built
The findings below come from our architecture, and they should generalize. These are ordinary, direct API calls. Still, it helps to know the setup so you can compare it to yours.
We run a custom orchestrator over managed third-party APIs (no self-hosted models in this dataset), and we run flagship models both directly through their providers (OpenAI, Anthropic, …) and through managed platforms (Bedrock, Databricks, …), so top models have more than 1 provider. That lets us compare serving paths and move work between them.
Our workloads are a mix: simple agent calls, deep reasoning, extractions, JSON and free text outputs. For a large fraction of calls we synthesize a large fact base into an answer, so large input and small to medium outputs. The analytics in this article hold input and output size constant within buckets (see appendix).
—
*Original article: [Building Reliable Customer-Facing LLM Workflows](https://contributor.insightmediagroup.io/) — based on production data from Databook, processing billions of tokens for the world’s largest enterprises.*# Cutting Steps Short: Why Interrupting LLM Calls Makes Systems More Reliable
## The Hidden Cost of Long Tails in LLM Workflows
When building production systems on top of large language models, the conventional wisdom says you should let each call run to completion — after all, the answer is coming, it’s just taking a little longer. But real-world data from enterprise workloads tells a very different story, and the counterintuitive move of cutting calls short at 20–30 seconds actually makes entire systems *more* reliable, not less.
## The Claim, and the Receipts
Here’s the move that sounds backwards: **cutting a step off at 20–30 seconds even when it might have answered perfectly a little later makes the system more reliable, not less.**
That isn’t a hunch. It’s true on paper — the math of heavy-tailed retries is unambiguous — and it’s true in the data. A scan of well over a million recent production LLM calls across enterprise workloads, drawn from real customer traffic, reveals just how strange a single call’s timing really is.
A typical longer-output call comes back in about a dozen seconds. But one in a hundred takes thirty seconds, sometimes a full minute or more — *for no reason connected to how much work it was doing.*
**FIGURE 3** – Real production data (1M+ calls, top-100 enterprise workloads, anonymized); 1s bins, capped at 90s. Model names are withheld on purpose. **This is not a leaderboard, and not a fair head-to-head:** different models run different workloads in the system, so the calls behind each curve aren’t the same task — the chart says nothing about which model is “faster.” What it *does* show: every model has a meaningful tail (note Model C — the quickest typical time, yet a long tail), and the *serving path* matters as much as the model — Model F via a managed API vs. direct is one model with two different tails. Model A shows free-form answer calls only; a separate, tightly-bounded structured-prefill workload on that same model is held out so it doesn’t split the curve into two artificial peaks.
That gap between the typical call and the slow one underlies much of this discussion. The rest of the article reviews what to do about it.
## Why the Clock Is Unforgiving
A workflow isn’t judged on its average. It’s judged against a deadline. On average, flows finish comfortably; however, outlier runs in long tails don’t. Those tail runs aren’t broken. They’d return a perfect answer a bit later, and on an internal run they would count as successes. On the customer’s side, every one of them is a failure. The entire tail of a latency distribution, however correct, becomes an addition to the failure rate.
That’s why the number that matters here isn’t average latency — it’s variance. A fast median buys nothing if the tail is long.
The second squeeze is **sunk cost**. The deeper a system is into a workflow, the more it has already spent: time, dollars, and TPM quota. A failure on step nine is far more expensive than the same failure on step two. Everything the workflow built gets thrown away, and there is less of the clock left to shift gears.
The system never restarts the whole workflow itself, but the customer will. If the system fails, the customer will almost certainly retry, starting the full flow again from the beginning. That compounds the problem. It burns more cost, more token budget, and the error budget on the SLA. And because the conditions that made the run fail usually haven’t changed, the retry has a similar chance of failing. Worse, it tends to happen during a high-TPM window — the worst possible time to pile extra load onto an already-strained system, and exactly when the odds of failing again are highest.
There’s a second multiplier, and it’s easy to miss. The first is the one from the opening: reliability compounds, so a chain of individually excellent steps can still come out a coin flip. But that failure is always told as a story about *correctness*: getting a wrong answer.
Here’s what you almost never hear about: **the exact same compounding happens on the clock.** Every step adds its own small chance of landing in the slow tail, and those chances stack. So the more steps that are chained, the more likely it is that *at least one* of them blows the deadline, even when every step is individually fast. That’s the multiplier this article is about, and it’s the one the literature leaves out.
## What an LLM Answer Time Actually Looks Like
The typical times in the chart above sit in a fairly tight band: every model finishes a typical call somewhere between eight and twenty seconds. The tails are not tight at all. One model’s 99th-percentile call comes in around 30 seconds, another’s past 80. Similar median, wildly different worst case. Promise a customer a median and you’re lying to the 1-in-20 and 1-in-100 calls in the tail, and a multi-step workflow hits those constantly. **A fast typical time is not a predictable one.**
The obvious objection is that the slow calls are just doing more work: bigger prompts, longer answers. They aren’t. Pin *both* the prompt size and the response length and the tail barely moves: within a single size bucket (work held fixed), p99 still runs **two to seven times the median** (Figure 4). The slowness isn’t about how much the call has to do — in production traffic it’s largely transient (queueing, scheduling, mid-stream contention, a provider hiccup), which is exactly what makes it worth interrupting.
**FIGURE 4** – “The tail isn’t the workload.” Each row fixes *both* prompt size and response size; the median climbs as the work grows, but inside every row the p50→p99 gap stays 3.8–6.7×. A dumbbell plot, deliberately not a distribution curve — same-size calls, wildly different finish times.
## One Slow Step Sinks the Whole Run
You’d think a workflow misses its deadline because many steps were each a little slow. It almost never happens that way. When a chain blows its budget, it’s usually *one* step that wandered into its tail while everything else behaved fine. Mathematically, a chain’s overrun is dominated by its single slowest step, not by the sum of many slightly-slow steps. That’s the property that makes the tail the thing to manage, and it’s the property that makes interrupting a slow call so powerful: you’re not giving up on a step that’s almost done, you’re cutting loose the one step that was going to ruin the whole run anyway.
## The Routing Hedge
This is where multi-provider routing becomes essential. Running several providers is what makes routing a hedge to a separate budget practical. With a single provider, fewer of these moves are available. When a call is cut short at 20–30 seconds, the system can route to a different provider rather than simply failing. This transforms what would be a hard failure into a recoverable event — but only if there’s somewhere else to send the call.
Note that if an architecture is self-hosted or on dedicated capacity, the tail may behave differently, and will warrant another approach. The transient nature of the slowness in managed API environments is precisely what makes the cut-and-reroute strategy effective.
## The Bottom Line
The data is unambiguous: heavy-tailed latency distributions in LLM calls are not about workload size, they’re about transient infrastructure conditions. A fast median is not a predictable system. And the compounding effect of chaining steps means that even individually reliable steps can produce unreliable workflows. Cutting calls short and routing elsewhere isn’t giving up — it’s the mathematically sound way to build reliable systems on top of unreliable tails.
—
*Original article: [Source – Insight Media Group Contributor](https://contributor.insightmediagroup.io)*# Cutting Early, Racing Later: Why Parallel Retries Beat Patient Waiting
**The total behaves like its maximum, not its sum.**2
This single insight reshapes how we think about latency in multi-step workflows. End-to-end timing is dominated by its single worst step, not by the accumulation of mildly slow ones.
That’s good news. You don’t need every step to be fast. You need to stop any single step from running away. Which is the cutoff.
—
## Cut Early, Then Race
If a wanders into its tail, waiting is the worst thing you can do — you’re spending your scarcest resource on your least likely payoff. So you give up early and try again *in parallel*: fire a fresh attempt and take whichever returns first. A fresh attempt rarely lands in the same pothole, so two fit inside the time one stuck call would have eaten — and the odds of *both* are tiny (if one is slow with probability *q*, both are slow with probability *q²*).3
**FIGURE 5** – The same longer step, waited out versus raced. Each dot is one production run of that step (top-100 enterprise traffic, anonymized); red marks the slow tail. Racing a second attempt and taking the first to return collapses the spread (std 6s → 3s, p99 roughly halved) for the price of extra tokens — the body barely moves, so you get the same typical speed with far less variance. A sequential re-draw on total time wouldn’t help: you’d pay the generation floor twice.
The median barely moves: about 10 seconds instead of 12. The tail does the opposite: the 99th percentile drops from roughly 60 seconds to 25, and the run-to-run spread is more than cut in half. You buy predictability for the price of some extra tokens.
That price is real, and it pushes back. Racing doubles token spending on that step, and tokens are a shared, capped budget. But the arithmetic is lopsided. Doubling *one* step costs tokens once, while blowing the deadline discards everything already spent and almost always triggers retries that re-run all *N* steps of the workflow — sometimes more.
The deeper into the flow you are, the more one-sided the trade.
—
## Match the Fix to the Failure
The direction your fallback takes has to match *why* the step is failing:
– **Slow for transient reasons** → re-draw, ideally in parallel. A fresh attempt escapes the stall. (A plain serial retry is weaker on a longer step — you’d pay the long generation time twice.)
– **Slow because the work is genuinely big** → don’t re-run the same call. Fall *down* to a faster model, or to an alternate path that reaches the same result more cheaply.
– **Wrong, not slow** → fall *up* to a more capable model. Speed won’t fix a bad answer; capability might.
—
## Cut on the Right Signal
Answer time has two phases.4 The wait for the **first token** is mostly queue and scheduling; the **generation** that follows takes the rest. Which phase carries the tail decides *what* to put the cutoff on.
For longer steps pressing against a deadline, the tail lives in **generation**. A slow queue is a small slice of a forty-second call; the spread that blows the budget is in the tokens. So cut on **total elapsed time**, or on tokens emitted so far against the time remaining — not on time-to-first-token.
Two signals are worth wiring in regardless:
– **No first token at all, past cutoff?** That’s stuck, not slow. Give up and hedge. A fresh parallel attempt gets newly scheduled and almost always wins.
– **Tokens flowing but slowly or producing a bad answer?** Hedge by *direction*, not just re-draw the same call: race a same-spec attempt for transient stalls, fall *down* to a cheaper/faster model for work that’s too big, and fall *up* to a more capable model for suspicious answers.
For any of these retries to work, the underlying behavior needs to be stochastic: parallel re-draws collapse transient tails and resource contention, but can’t shortcut a deterministic slow answer. That’s why hedging logic must first diagnose *why* a step is failing — the fix depends on the fla—
### Sidebar — The Math, Briefly
**Compounding.** Just the arithmetic of independent steps: *n* steps each succeeding with probability *p* give *pⁿ* end-to-end. At *p* = 0.95, ten steps ≈ 60% and twenty ≈ 36% — multiplication, no modeling. Independence simplifies (shared capacity correlates real steps) but it’s the conservative, illustrative case.
**The single big jump.** For independent subexponential-distributed steps, the tail of the sum is just the sum of the tails — `P(ΣX_i > t) ≈ Σ P(X_i > t) ≈ P(maxᵢ X_i > t)` as *t* grows. In words: a chain overruns because *one* step hit its tail, not because many were mildly slow.2
**Hedging.** Fire *n* independent attempts and take the first good one: if one attempt fails with probability *q*, all *n* fail with probability *qⁿ*. This arithmetic doesn’t care *what* “fail” means — deadline blown, hard error, or wrong answer all buy down the same way. For timing specifically it also shrinks spread: capping each step’s tail shrinks the whole chain’s variance. All of it rests on independence (fresh draws, fresh queue).3
—
2 The “single big jump” principle — for heavy-tailed (subexponential) distributions, the tail of a sum is dominated by the tail of the maximum term, not by accumulated mild deviations.
3 Hedging via parallel redundancy — since failure probability compounds geometrically (*qⁿ*) across independent attempts, even a few parallel draws dramatically reduce the chance of total failure.
4 Two-phase latency decomposition — total response time splits cleanly into time-to-first-token (queue/scheduling) and generation time (token emission), with the dominant failure mode depending on output length.
7 Variance collapse via racing — collapsing the spread through parallel draws, while preserving median performance, is the foundation of predictability in heavy-tailed inference workloads.# Cut Early, Fall Back: Using Latency Cutoffs and Hedging to Tame the Tail in LLM Workflows
## The Problem with the Slow Tail
Every LLM workflow has a tail. It doesn’t matter how fast a model normally answers — occasionally, a step just drags. Maybe it hangs somewhere in the service, maybe it catches a cold, maybe it’s a fundamentally harder instance. Whatever the reason, the latency distribution has a long right tail, and that tail is where your workflow lives or dies.
Most teams handle this by setting a generous timeout and hoping. That’s the wrong approach. When a step is going to fail, it’s far better to *decide* it’s going to fail — on a timer you choose — and redirect the workflow while there’s still time and budget left. This is the idea behind latency-based cutoffs and hedging, and done well, it has an outsized effect on the cost curve of many LLM workflows.
The mechanism is simple at a high level: measure how long each step’s good answers take, then cut any step that runs long and trigger a fallback before the wall clock becomes the problem.
There are three distinct actions under this umbrella, each with different trade-offs:
1. **Latency-based timeouts** — a simple clock that interrupts a step unless it finishes in time.
2. **Hedged requests (racing)** — fire two calls at once and take whichever finishes first.
3. **Length cutoffs** — cap a step’s answer length, because in most workflows a partial answer beats a stalled one.
The rest of this is about how to set these up so they help more than they hurt.
## Latency-Based Timeouts: The Cutoff Clock
The simplest intervention. For each step in your workflow, attach a hard deadline measured from when the step fires. If the result hasn’t come back by the deadline, the step is marked failed, the workflow branches to a fallback, and the slow call is abandoned (as much as you can abandon it — more on that shortly).
The word *measured* matters. Don’t guess the cutoff. Run the workflow many times, record the latency of every step that completes *successfully*, and pull the P95 off that distribution. That number — the time by which 95% of good answers are already back — is your starting cutoff. You tune it down from there depending on the role of the step in the workflow (more on that later).
The number is usually a lot tighter than people expect. A step that takes 1–20 seconds on most runs and occasionally hangs at 30–60 gets cut at 25 or 30. You accept that you occasionally cut a call that *would* have succeeded, and in exchange you almost never wait 60 seconds only to fail anyway.
This is why precomputing cutoffs from measured latencies is better than inferring deadlines dynamically from the workflow graph. Dynamic inference is a neat idea in theory — you have a total time budget, you subtract the expected cost of future work, and you set each step’s deadline based on what remains — but in practice it relies on a cost estimate that’s hard to compute and static latencies that don’t reflect real-case difficulty. Measured cutoffs ground you in what the step actually does.
The downside of hard cutoffs is obvious: you cut steps that would eventually have answered correctly. That cost is real. The “design the dependencies out” guideline above (from *How our system is built*) is partly motivated by this — breaking the workflow into smaller, more independent steps makes a timeout less costly, because you’ve lost only that small step and have other ways to proceed.
## Hedged Requests (Racing): Escaping the Stall with a Second Call
If you’re willing to spend tokens, you can do better than just giving up. A hedged request works like this: you fire your primary call. If it has not returned by a measured threshold — typically a fraction past the P95 — you fire a *second* call in parallel and take whichever finishes first.
The second call is routed through a *different model or a different provider*, not the same one. That’s important for two reasons: it gets an independent draw on latency (the stall was local to one provider, not a property of the prompt), and it keeps your within-budget rates independent (more on that below).
Hedging is especially good at catching the *fast failures* — the long tail of slow answers is partly made of calls that will be slow because the provider is congested, and a second call sent to a separate provider escapes that congestion rather than joining the same queue.
The racing also catches a failure mode a timeout silently passes over: a step that returns *quickly* but returns junk — empty, truncated, or unparseable. A fast wrong answer still wins the race! The hedge doesn’t directly reveal it. For that you need the next idea.
### Strengthen the Race with a Fast Draft Model
One useful variation: route the second, faster call to a cheaper, faster model instead of a full-quality one. You can still race the original call against it, and the draft model will often finish first. Now you have an answer quickly, and if the primary call later returns a better answer, you can upgrade.
But if the primary call is slow or fails entirely, you’ve got a usable answer from the draft model that you wouldn’t have had. It trades some quality on the races it wins for massive tail improvement, and the fast model is so much cheaper that the budget hit is small.
### Strengthen the Race with a Post-Call Validator
This addresses the junk-answer problem. After the race finishes, immediately validate the expected output. For any step that’s supposed to return a specific shape — a structured object, a list, a properly formed action — the cheapest such check is a strict schema validation right after the call. Parse the result against the expected object; if it fails, treat it exactly like any other failure and fall back.
It catches a meaningful fraction of bad answers before they reach the next step and silently corrupt downstream work. The cost is trivial compared to the call itself, and it’s the only way to catch the fast failures that latency management alone misses.
## Length Cutoffs: Don’t Run On Forever
This one’s simple to state: some steps have a natural maximum answer length, and you should enforce it.
You can often pick a model setting that caps the response at a number of tokens. Pick a length that comfortably covers 99% of your successful runs (measure it), enforce it, and don’t re-run on truncation. A truncated result that came back at full speed is usually unusable; you just get to spend more tokens on a model that can’t follow the length instruction. Fall to a faster model instead.
## The Catch: Hedging Spends the Budget You’re Shortest On
Racing has an awkward property. The tail is worst when the system is busy. And “busy” is exactly when your tokens-per-minute budget has the least room left. So the one move that fixes the tail wants to spend tokens at the precise moment they’re hardest to come by. Do it blindly and you get a pile-on: slow calls trigger hedges, hedges add load, load makes everything slower, more calls cross their cutoff. A latency problem becomes a rate-limit problem.
Two facts make this less forgiving than it first looks. The cost is committed the instant you fire the second call. Cancelling the loser frees *your* connection, but the provider keeps generating, and billing, the abandoned attempt. There’s no clawback, so all the control has to live at the decision to hedge, not after. And you usually can’t see how much budget is left. Estimating it is possible but involved, so any scheme that “eases off as the quota fills” is hard to run in practice.
What works in practice is cruder and more structural:
– **Send the hedge somewhere with its own budget.** Token limits are per-model and per-provider, and most of us run more than one (as noted in *How our system is built*). Routing the retry to a *different* model or provider gets a separate quota *and* an independent draw. The same move that escapes the stall also avoids spending the scarce budget twice.
– **Keep hedges rare by construction.** This is what the precomputed cutoffs already buy you: with the threshold set at each step’s measured P95, a hedge fires only on the slow minority, so the extra spend stays small with no runtime accounting at all. (Same cutoffs as the next section, no new machinery.)
– **React to the signals you actually get.** You probably can’t read headroom, but you can read 429s and climbing latency. Treat those as the cue to hedge *less* and cut *later*, not more.
– **At real saturation, stop hedging.** Once the provider is already returning rate-limit errors, more attempts only deepen the hole. Downshift to a smaller, cheaper model or shed the work instead.
One lever we haven’t built, and offer only as a direction: an explicit global cap that holds hedged calls to a small fraction of total traffic, independent of the per-step decisions. It’s the principled backstop the tail-at-scale work points to; we set conservative cutoffs instead and haven’t needed it, but at higher hedge rates that’s where we’d go next.
## Sidebar — The Cheap Moves You Make First
Cutoffs and hedging are insurance. You buy less of it if the workflow is built well to begin with. The defaults that fire on *every* request, before any reactive trick:
– **Parallelism by design.** Lay the flow out as a dependency graph and run every step the moment its inputs exist. Then go further — *design the dependencies out.* Fewer dependencies means more steps are leaves, and a leaf can fail cheaply without taking the rest of the graph down.
– **Don’t call the model at all when you don’t have to.** The most reliable call is the one you don’t make — use code, lookups, and validators wherever the work doesn’t actually need a model.
– **Mix models per step, not per workflow.** Fast and cheap where it’s enough; capable where it isn’t.
– **Cache the deterministic parts.** Don’t pay an LLM twice for an answer that can’t change.
The point here: spend your reliability budget on structure first, so the clock work has less to fix.
## When Do You Actually Pull the Trigger?
The cutoff is a knob, not a constant. How hard you turn it comes down to three plain questions about each step:
1. **How much does the answer need this step?** Nice-to-have: let it go. Must-have: protect it.
2. **How much is waiting on it?** If nothing depends on it, let it run to the deadline. If half the workflow is queued behind it, finish it sooner, and make sure it’s *right*, because a wrong answer here poisons everything downstream.
3. **How much time is left?** Plenty: retry calmly. Almost out: cut fast and fall back.
The more a step is must-have, load-bearing, *and* short on time, the earlier you fire the backup and the more you’ll spend to hedge it. An optional, terminal, early step gets none of that. (“Early or late in the flow” was never the real axis. It was a proxy for how much still depends on this step.)
And you don’t guess the number. You run the workflow many times, measure each step’s latency curve (P95), and set the cutoff from that curve. Below the step’s worst case, weighted by the three questions. A step that usually answers in 20 seconds gets cut at 30, even though it might have succeeded at 60.
## Why Almost Nobody Does This
This isn’t hard. It’s nuanced, and most teams don’t have the engine for it.
The popular workflow tools, the Airflows and Temporals, were built to make pipelines *durable*: retry, resume, don’t lose state, and they’re very good at it. Their timeout advice follows from that goal: set a per-step timeout *longer* than the slowest run and retry until it succeeds. That’s the right instinct when the job is to *durable completion*, and it’s **exactly the wrong advice** when the job is to finish *in time*. Your workflow engine will happily retry a step many times; it has no notion of a step’s measured typical time and downstream implications, so it can’t cut early and switch.
—
*Original article: [Cut Early, Fall Back: Using Latency Cutoffs and Hedging to Tame the Tail in LLM Workflows](https://example.com/original-article)*# Predictable Completion Times Trump Raw Speed in AI Systems
Building reliable, customer-facing AI agent workflows isn’t about chasing low latency — it’s about minimizing variance. A system with a predictable completion time will always beat one with fast median responses but devastating long tails.
## The Statistical Reality of Model Latency
LLM latency isn’t Gaussian. It follows a lognormal subexponential distribution, meaning the tail of a sum of calls is dominated by a single big jump — a principle established in heavy-tailed distribution theory and confirmed in production data. In practical terms, this means one slow step in a multi-step workflow dominates the total elapsed time, even when every individual call looks reasonable at the median.
This matters enormously for compound reliability. Ten steps each at 95% success rate yield only about 60% end-to-end reliability; twenty steps drop to around 36%. Each additional step erodes the overall probability multiplicatively.
## Why Retry Logic Isn’t Enough
Traditional distributed systems tools — retries, deadlines, hedging — aren’t flawed; they’re simply designed for cases that don’t assume a short, non-resumable budget where the right action at the cutoff isn’t trying the same call again, but instead switching to a *faster alternative*. The principle is the same, just applied in an inverted direction.
When racing two parallel attempts against production data, the p99 latency on longer steps dropped by more than half — from roughly 60 seconds down to 25 seconds. The reason is mathematical: if each independent attempt is slow with probability *q*, two parallel attempts are both slow with probability *q²*, and *n* attempts with *qⁿ*. Since the tail in longer calls sits in the generation phase rather than the time-to-first-token window, cutting total elapsed time — not just prefill — is where the wins come from.
Standard inference guidance splits latency into “time to first token” (queue plus prefill) and per-token generation. For the workloads measured, the generation phase was where the tails lived.
## Designing for the Bound, Not the Median
A critical insight: **a predictable completion time beats a fast one with a long tail.** Low variance beats low latency. You can’t promise a customer a median response — you can only promise a bound. Every engineering decision discussed here serves that bound.
The strategies include cutting early, hedging requests, racing parallel calls, and designing out dependencies. Each trades a modest amount of average speed for substantially reduced variance — giving up the right tail to buy the left.
## The Tail Isn’t Just Bigger Calls
A natural objection is that the slow tail simply reflects larger outputs — more tokens to prefill, more to generate, so naturally slower. The data contradicts this directly. Within-cell p99/p50 tail ratios stayed flat at roughly 2–4× across every output-size bucket — they didn’t climb as the work grew.
Most decisively, the leftmost bucket contained calls emitting at most 50 output tokens. Generation time physically could not vary by more than about a second, yet the tail ratio there was still approximately 3.5×. There is no size variable large enough to produce that spread. The residual variance is transient: queueing, scheduling, a momentary provider hiccup — exactly the kind of thing a fresh attempt escapes.
The two figures — volume-weighted averages and per-call envelope values — reflect the same data viewed two ways. The averages demonstrate the tail doesn’t scale with work; the envelope shows how wide it gets on any given call.
## Why This Matters for Agentic Workflows
In a customer-facing agentic workflow, reliability *is* the product. The craft isn’t owning a bag of retries and fallbacks — those are table stakes. It’s deciding, per step, *whether* to hedge and *when* to give up, informed by the constraints and the measured behavior of your own system.
Distributed systems fundamentals apply directly: work from a deadline budget, match each timeout to measured latency. The extension is applying these tools to a case they don’t traditionally assume — a short, non-resumable budget where the right move at the cutoff is a faster alternative, not the same call again.
The slow tail in production traffic is largely transient, which is part of why hedging and racing work so well. A managed API serving path versus a direct one affects predictability significantly, so the analysis focuses on serving paths and model families rather than ranking models in the abstract.
## Bottom Line
For enterprise AI infrastructure powering go-to-market teams at scale, the path to reliability runs through variance reduction, not raw speed optimization. Race when you can, cut when you must, and always design around a bound your stakeholders can depend on.
—
## APPENDIX
### About the Author
Frank Wittkampf is Head of Applied AI Engineering at Databook. His team architects, builds, and operates a fully custom AI stack including deep reasoning, an agentic workflow engine, AI asset generation, agentic harnesses, knowledge base and context graph, AI pre-processing, multi-tenant AI configuration management, and more. This AI infrastructure powers the GTM teams of top Enterprise companies like Microsoft, Salesforce, Amazon, Databricks, and many others.
### A Note on the Data
The latency figures come from June 2026 anonymized production traffic across enterprise customer workloads — roughly 1.2 million LLM calls over a 30-day window. These are direct calls to managed third-party APIs, which is part of why the slow tail is largely transient. The numbers describe longer calls (output ≥ 600 tokens), since those are the ones that actually press against a deadline. A “tail ratio” (p99/p50) holds call size fixed within a bucket unless stated otherwise. Durations were bucketed in one-second bins with a 90-second ceiling truncating only the last ~0.2% of longer calls.
—
**Notes & Footnotes**
¹ Ten steps at 95% each ≈ 60% end-to-end; twenty ≈ 36% (assuming independence).
² The subexponential class produces the “single big jump” principle — one slow step dominates the sum. Intuition for why, not a plug-in formula.
³ Classic hedged-request results (Dean & Barroso, “The Tail at Scale,” CACM 2013); ICML 2026 work on choosing between serial, parallel, and hedged execution from measured latency curves.
⁴ Standard split in inference: time-to-first-token versus per-token generation. See Agrawal et al., *Taming the Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve* (arXiv:2403.02310, 2024).
⁵ Temporal activity timeouts are designed to finish eventually, including retries — hence Start-To-Close set above the slow tail.
⁶ Google SRE, gRPC deadlines, and Spanner all propagate a total budget and drop work that can no longer help the caller. The same principle extends here to a synchronous, non-resumable customer budget.
