# Why “Deterministic” LLM Outputs Still Diverge
**Every large language model sits on top of billions of floating-point additions. That fact alone is enough to explain why two identical, temperature-zero queries can produce different answers.**
—
## The Puzzle of the 80 Different Answers
Imagine sending the exact same prompt to a model one thousand times, asking it to behave perfectly deterministically. You have set every knob to zero randomness. Surely the output should be a single, unwavering string of text.
That is what most practitioners would assume. Yet in a revealing experiment, researchers observed that out of one thousand repetitions, eighty distinct completions emerged — despite temperature being set to absolute zero, where the model is supposed to greedily pick the single most probable token at every step.
Even more striking was the pattern of divergence. The first hundred tokens were identical across all one thousand runs. Then, at a single position, the majority of completions chose one path while a small minority veered off onto another. After that split, the two groups diverged further with each successive token.
This behavior raises a deep question: if the mathematics is deterministic, why does the answer depend on seemingly invisible factors like who else is using the server, which GPU kernel gets scheduled, or how many requests are batched together? And how can we predict when — and how often — such surprises will occur?
—
## Floating-Point Arithmetic Is Not Exact
The root cause begins with the hardware itself. When computers perform addition using 32-bit floating-point numbers, the result depends on the order in which values are summed. This is because floating-point has limited precision, and rounding errors accumulate differently depending on sequence.
Consider a simple example: adding 0.1, 0.2, and 0.3 in one order might yield exactly 0.6, while the same three numbers in a different order might produce something like 0.6000000000000001. Both are “correct” within the tolerance of the format, but they are not bit-identical.
A neural network forward pass consists of billions of such additions. Each layer involves matrix multiplications, normalization steps, and nonlinear activations — all built from floating-point sums. When the hardware chooses a particular order for these operations, it shapes every intermediate value and ultimately every logit score that the model uses to select its next token.
## Why Batch Size Controls the Outcome
A common explanation for non-deterministic outputs blames the chaos of GPU thread scheduling — the idea that thousands of parallel threads finish their work in unpredictable order. But careful investigation has shown this is not the primary driver. Individual kernels, when given identical input and running in isolation, reproduce the same bit patterns every time.
The real culprit is subtler. Many GPU kernels are not **batch-invariant**, meaning their internal reduction order changes depending on the total number of inputs being processed simultaneously. On a shared inference server, the batch size fluctuates with incoming traffic — sometimes you share a GPU with one other user, sometimes with ten. Each different batch size subtly reorders the floating-point reductions inside the model’s layers.
The result is that your output depends not only on the prompt and the model weights, but also on the strangers sending requests to the same server at the same moment. This effect is not unique to GPUs; it would emerge on any architecture where batch-dependent reduction orderings exist. Researchers have confirmed that when batch-invariant kernels are used, every single one of those one thousand repetitions produces the identical answer.
There is a trade-off, though. The batch-invariant, deterministic version of a serving stack ran roughly twice as slowly as the standard optimized version in benchmarks, making it an impractical default for production systems that prioritize throughput.
## Where the Splits Actually Happen
Not every token position is equally dangerous. The overwhelming majority of positions carry a comfortable gap between the highest-scoring token and the runner-up. At these positions, the floating-point noise present in any real computation is far too small to overcome that margin. The model’s choice is rock-solid.
The splits occur exclusively at **near-tie positions** — places where the top two candidate tokens have nearly identical logit scores, perhaps separated by only a few hundredths of a logit unit. At such positions, the tiny numerical errors introduced by finite-precision arithmetic can tip the balance, and the model will happily select a different token.
This means that the probability of a token choice flipping is not uniform across a sequence. It concentrates on a small number of fragile positions. A completion might run through dozens of tokens without a single disagreement, and then split sharply at one knife-edge position where the top two logits sit almost exactly on top of each other.
## A Simple Formula for Flip Probability
The mathematics of this phenomenon can be captured compactly. If we define the gap as the difference between the top two logits at a given position, and we assume that floating-point noise adds a small random perturbation to that gap, then a flip occurs whenever the noise is large enough to overcome the gap and push the ranking the other way.
Averaged across all positions in a sequence, the per-token flip probability can be approximated by:
**p ≈ f(0) · E|Δ| / 2**
Here, **f(0)** represents the density of gaps near zero — essentially, how crowded the model’s near-tie positions are. The term **E|Δ|** captures the expected magnitude of the numerical error on those gaps. The factor of one-half comes from averaging the random, symmetric noise over the narrow region that matters.
This formula cleanly separates two concerns. The first factor, f(0), is a property of the model architecture, its training, and the specific text being generated. The second factor, E|Δ|, is a property of the hardware, the numerical precision being used, and the implementation of the inference kernels. Understanding which factor dominates gives practitioners actionable insight: sometimes the problem is the model (too many near-ties), and sometimes it is the serving stack (too much noise per kernel).
## Testing the Theory on Real Models
To validate this formula, researchers ran experiments on GPT-2, comparing inference passes performed in lower-precision brain floating-point format against higher-precision standard format on a laptop. The comparison acts as a stand-in for the kinds of precision-induced errors that occur in real serving environments.
The density of gaps near zero was measured at approximately 0.83 per logit, meaning nearly every position has at least a modest degree of near-tie behavior. The average numerical error on those gaps was around 0.02 logits. Plugging these values into the formula predicted a flip rate of roughly one percent — a number confirmed almost exactly by direct measurement across 256 input sequences.
Further validation came from injecting controlled amounts of synthetic noise into the model’s real logits and measuring the resulting flip rate. The observed behavior tracked the formula’s predictions across a wide range of noise levels, reinforcing confidence that the framework captures the essential dynamics.
## From Individual Tokens to Whole Completions
The per-token flip probability tells us about single positions, but what practitioners actually care about is whether an entire completion stays the same between runs. The answer follows a survival logic: if each token independently carries a small probability of splitting, the chance that a completion of a given length remains fully identical decays exponentially.
However, real text is not uniform in its fragility. Some prompts contain many knife-edge tokens packed tightly together; others flow through long stretches of wide-margin, stable positions. When prompts vary in their inherent fragility — which they always do — the distribution of completion agreement times becomes heavier-tailed than a simple exponential decay would suggest.
This means that the average flip rate alone is a poor predictor of practical consistency. A model might produce identical answers ninety-nine times out of a hundred, and then deliver a starkly different response once, all because a single near-tie token triggered a cascade of divergence. The risk is lumpy and concentrated, not evenly distributed.
## Practical Guidance
### Predicting Your System’s Flake Rate
If you know the per-token flip probability for your particular stack, you can estimate how often complete answers will differ. For a completion of length L tokens, the probability of at least one disagreement is approximately:
**1 − exp(−p · L)**
A flip rate of one per thousand tokens applied to a 500-token response yields roughly a 39% chance of any two runs diverging. By measuring gap density and numerical error on your own model and hardware, you can estimate this rate before deploying evaluation pipelines or safety-sensitive applications.
### Rethinking Exact-Equality Comparisons
Automated evaluation pipelines that check for byte-perfect string matches between two temperature-zero runs are fundamentally fragile. They will produce false negatives whenever a near-tie token flips. The healthier approach is to compare with a tolerance, or to grade responses semantically rather than symbolically. If you control the inference infrastructure, batch-invariant kernels eliminate this source of variance at the cost of some throughput. If you do not control it, you should assume that two identical API calls can and will occasionally return different strings.
—
## Frequently Asked Questions
**Q: If temperature is zero, shouldn’t the output be completely deterministic?**
A: In pure mathematics, yes. In actual computer arithmetic, no. Floating-point addition is not associative, and the order in which a GPU performs billions of these operations depends on factors like batch size and kernel scheduling. These tiny rounding differences can cascade through a network and ultimately change which token has the highest score.
**Q: Is this a bug that will be fixed in future models?**
A: Not exactly. This is a consequence of how modern hardware performs computation at scale. It can be mitigated — for example, through batch-invariant kernels — but such fixes typically carry a throughput cost. The phenomenon is a feature of the computational substrate rather than a flaw in model design.
**Q: How dangerous are these flips in practice?**
A: It depends on the application. For creative writing or casual conversation, a flip that swaps near-equivalent phrasings is harmless. For structured data extraction, code generation, or safety-critical evaluations, even a single flipped token can produce a materially different and potentially incorrect output. The key insight is that flips happen at predictable frequencies and can be measured.
**Q: Can I reproduce the original experiment on my own hardware?**
A: Yes, the core experiment is straightforward. Run the same prompt hundreds of times at temperature zero, record the outputs, and count the number of unique completions. The specific split patterns will vary by model and hardware, but the underlying phenomenon is universal across modern transformer architectures.
**Q: Does this mean I should never trust temperature-zero outputs?**
A: You should trust them as approximate answers, not exact ones. For most practical purposes, the outputs are close enough. But if your pipeline requires bitwise determinism — for example, in a reproducibility audit or a legal evidence chain — you need to either control the inference environment carefully or use semantic comparison rather than exact string matching.
**Q: Is the flip rate the same for all models?**
A: No. Models trained with different architectures, quantization schemes, or training data will have different gap distributions near zero. Larger models with more parameters may have more or fewer knife-edge positions depending on how their logits distribute. The formula provides a way to measure the flip rate for any given model and serving stack.
—
## Conclusion
Temperature-zero generation in large language models is deterministic in theory and messy in practice. The divergence we observe between repeated runs traces back to the same source that affects all numerical computing on modern hardware: the finite precision of floating-point arithmetic and the order-dependent nature of parallel reduction operations.
The good news is that this behavior is not random noise without structure. Near-tie positions concentrate nearly all the risk, and the flip probability at each position can be estimated with a simple, interpretable formula grounded in the density of those ties and the magnitude of the numerical error on your hardware. With this understanding, practitioners can make informed decisions about how to evaluate, compare, and deploy LLM systems — knowing exactly how much to trust the illusion of determinism.
The path forward involves either engineering inference pipelines that eliminate batch-dependent variation, or designing evaluation methodologies that are robust to it. Both approaches are viable, and the choice depends on the application. What matters is that the mystery of why identical inputs can produce different outputs has been demystified — it is a quantifiable, predictable consequence of computing at scale.
Thank you for reading
