# When Valid JSON Becomes a Liability: The Hidden Cost of Constrained Decoding
Structured output from language models has transformed how production AI systems operate. What once required fragile prompt hacks and hope now runs through finite-state grammars that guarantee every token sequence is parseable. Teams migrated to these methods with good reason — the alternative was an endless loop of retries, regex fixes, and broken pipelines.
But there is a cost hiding inside that guarantee, and most teams haven’t noticed yet.
## The False Promise of Structural Correctness
When a language model is forced to produce output that conforms to a strict grammar, every single field will be the right type. Strings will be strings. Numbers will be numbers. Arrays will be arrays. Enum values will always be members of the declared set.
That is where the promise ends.
Schema validation answers one question: does this output have the right shape? It does not answer the far more important question: is this output actually correct? A perfectly shaped JSON document can contain values that are completely wrong, fabricated, or internally inconsistent — and every validator will approve it without complaint.
This is not a theoretical concern. In real classification pipelines built on structured output, wrong answers arrived dressed in perfect JSON roughly once every twelve runs. Each one passed every downstream check because those checks tested structure, not meaning. The errors went undetected for weeks.
## What Forcing a Format Actually Costs
The trade-off is straightforward and rarely discussed. When a model knows it must stay inside a rigid output format, part of its reasoning capacity goes toward format compliance. Less capacity remains for getting the actual answer right.
Empirical measurements across open-weight models show that enforcing structured output formats causes accuracy drops between three and nine percentage points in general tasks. On reasoning-heavy workloads — particularly mathematical problem solving — the gap widens substantially, exceeding fifteen percentage points in some cases. The format itself becomes a tax on accuracy, and most production pipelines measure neither the tax nor the cost it exacts.
The stricter the formatting constraints, the more reasoning quality degrades. This relationship holds consistently across model families and task types, which means it is not a quirk of a particular architecture or training approach. It is a fundamental tension between correctness of form and correctness of content.
## Five Ways Schema-Valid Output Can Still Be Wrong
Production experience reveals five recurring failure patterns, each of which produces output that is structurally flawless yet substantively broken.
### 1. Valid Values in the Wrong Context
A priority field with an enum of low, normal, high, and urgent will always return one of those four values. The grammar cannot tell the model which one is appropriate for a given input. The result is a valid enum token selected for the wrong reason — urgent on a routine request, low on something critical — with no mechanism in the schema to distinguish the correct choice from the incorrect one.
### 2. Invented Data That Sounds Plausible
When the schema demands a value and the input does not support one, the model fills the gap. Show it a photograph of an elephant and ask for a receipt expense report, and it will produce a fully formed, schema-valid document with fabricated vendor names, amounts, and dates. The constrained grammar leaves no room for the model to say “I cannot answer this.” It must produce something, and it does.
### 3. Fields That Are Fine Alone But Impossible Together
Individual field validators check each piece in isolation. They never compare fields to each other. A sentiment analyzer might return a positive label with a confidence score near zero — both values are technically valid, but together they contradict each other. A date range might have an end date before the start date. Each field passes its own check; the record as a whole is nonsensical.
### 4. A Creeping Drift Toward Safe Defaults
Over time, models under constrained decoding converge on high-probability, generic values. Medium confidence instead of context-specific scores. General categories instead of precise ones. The field distributions flatten, the entropy drops, and the pipeline quietly stops producing useful variation. Nothing looks broken at the structural level, but the output has stopped being informative.
### 5. Fabricated Array Entries
Empty arrays are grammatically awkward under constrained decoding. The model’s generation probabilities favor object-producing token paths over the path that leads to `[]`. The result is arrays that are never empty — instead of reporting zero matches, the model invents three, five, or a dozen phantom entries that your downstream logic happily processes as real findings.
## Why Adding More Validation Rules Is Not the Answer
The natural reaction to these failures is to write more validators. For known patterns, this works. A cross-field check can catch date ranges where the end precedes the start. A custom rule can flag sentiment scores that contradict their labels. But this approach has a fundamental limitation: it only covers failures that have already been discovered.
Semantic correctness is an open-ended problem. The space of wrong answers is infinite and the model will find new ways to be wrong that no existing validator anticipated. Each new rule closes one known gap while leaving every unknown gap open. In production, the rate at which models discover new failure modes outpaces the rate at which teams can write new rules.
There is also a deeper structural issue at play. When a schema requires a value, the model provides one — regardless of whether the input justifies it. There is no standard way for the model to communicate uncertainty, abstain, or indicate that a field does not apply. The schema acts as a forcing function that produces confident wrong answers with the same outward certainty as correct ones.
## A Three-Layer Strategy for Reliable Structured Output
Treating schema compliance as the entirety of your quality assurance is a false sense of security. A more robust approach layers three types of checks, each covering what the others miss.
**Layer One: Schema and Structural Validation.**
Keep your Pydantic models, JSON Schema definitions, and Zod types. They solve the syntax problem well — catching missing fields, wrong types, and invalid enum values. This is the necessary foundation, and no production system should skip it.
**Layer Two: Semantic Validation and Distribution Monitoring.**
Write cross-field constraint functions that encode business logic: if the sentiment is positive, the score must exceed a threshold. Build entropy monitors that track value distributions over time and alert when a field collapses to a narrow set of defaults. Periodically sample outputs from ambiguous or edge-case inputs and have domain experts review them. This layer catches most of the five failure modes because it checks meaning rather than shape.
**Layer Three: Uncertainty Surfacing.**
Add optional confidence or reliability fields alongside every extracted value, giving the model a way to express what it does not know. For high-stakes fields, use a second model call to judge whether the extracted value is supported by the input. This adds latency, but for fields where a wrong answer carries heavy consequences, the cost is justified.
Most teams operating today are running on Layer One alone. Moving to Layer Two catches the majority of silent failures. Layer Three is appropriate for the specific fields where accuracy matters most.
## Three Statistical Signals That Something Is Wrong
Individual output inspection will not reveal these problems. The warning signs are aggregate and statistical.
Watch for **dropping output entropy** — if a field that should vary across inputs starts clustering around one or two values, the model is defaulting. Track **empty-array rates** — if an array that should sometimes be empty never is, the model is fabricating entries. And pay attention when **downstream business metrics shift** without any change to the model version, prompt, or schema. Structural compliance can remain perfect while semantic accuracy erodes, and that mismatch is often the first indication that something has gone wrong.
## The Right Way Forward
Constrained decoding is the right default for parse reliability. Every production pipeline that depends on structured output should use it. But parse reliability is only one dimension of output quality. The path to trustworthy AI systems requires building the semantic checks that grammar enforcement was never designed to provide.
Schema validation tells you the data is shaped correctly. The harder, ongoing work is ensuring the data is actually correct. That work lives outside the grammar and inside the domain logic, distribution monitoring, and uncertainty handling that surround it.
—
## Frequently Asked Questions
**Does this mean I should stop using constrained decoding?**
No. The parse reliability benefits are real and substantial. The recommendation is to keep using constrained decoding while adding layers of semantic validation on top of it. Unconstrained generation with post-hoc parsing is an alternative worth evaluating for accuracy-sensitive workloads, but it introduces its own trade-offs around latency and guaranteed parse success.
**Can I write validators that catch all five failure modes?**
You can write validators for known failure patterns, and you should — cross-field checks and distribution monitors are essential. But semantic correctness is an open-ended problem. New failure modes will emerge that no validator currently catches. The goal is not to write a validator for every possible error but to build a monitoring and detection system that catches anomalies as they arise.
**How do I start building these checks in an existing pipeline?**
Begin with entropy monitoring on the fields most critical to your business logic. A simple weekly plot of value distributions can reveal distributional collapse within hours. Next, add cross-field constraint checks for the most common contradictions in your domain. Finally, introduce periodic human audits on edge-case inputs — this is where confident fabrication is most likely to surface.
**Is the accuracy gap between constrained and unconstrained generation consistent across models?**
The direction of the gap is consistent: forcing structured output reduces accuracy. The magnitude varies by model family, task type, and how strictly the output is constrained. Math and reasoning tasks tend to show larger gaps than simple extraction tasks. The practical implication is that the stricter your format requirements, the more important your post-hoc validation becomes.
**What is the best way to handle fields where the model has no basis for an answer?**
The current best practice is to surface uncertainty explicitly. Adding a confidence or reliability field alongside every extracted value gives the model an honest output channel. For the highest-stakes fields, a second-pass LLM-as-judge verification step can catch failures before they propagate through the pipeline, at the cost of additional latency.
—
Thank you for reading
