**Smart Model Selection in Enterprise Document Intelligence: A Cost-Safe Cascade**
Modern document-intelligence pipelines often reflexively call a large hosted API to extract typed fields such as amounts, dates, and coverage limits. While this works, the bill quickly becomes the largest line in the pipeline’s running cost. In reality, many fields are plain lookups well handled by smaller models, and paying flagship prices for all of them is unnecessary. The smarter approach is not “always big” or “always small,” but rather **choosing the model by criteria, verifying the answer, and escalating only when verification fails**—the model-selection loop that defines an LLM cascade.
This article complements the *Enterprise Document Intelligence* series and drills into the generation brick, answering: which model should run the call, and what happens when the cheap one is not good enough? It builds on Article 6C (dispatch), Article 8C (validation), and Article 10 (adaptive parsing), forming a consistent cheap-first, escalate-on-failure shape for extraction workloads.
All recommendations below are backed by a real benchmark: a field-extraction task across hundreds of typed fields and dozens of documents, plus a focused sweep of twenty local models against one hosted flagship, all run at temperature 0 with JSON output. No document-, field-, or figure-level data from that benchmark appears in the narrative.
—
### 1. Two angles: cost, and the loop
There are two interconnected reasons to rethink model selection.
– **The cost angle:** Flagship models cost an order of magnitude more per call than small local ones. When extraction jobs involve thousands of field-level calls—or a document assistant answers thousands of questions per day—the model bill dominates runtime cost. Most calls do not need a flagship; routing simple calls to cheaper models leaves money on the table when unused.
– **The loop angle:** Cutting cost by naively forcing a small model everywhere risks broken or unreliable output. If answers are not validated, wrong values ship. The solution is not “use the big model,” but rather **run the cheap model first, validate, and escalate only on failure**. This bounded loop ensures cost savings while preserving correctness. In the literature, this pattern is known as an **LLM cascade**.
—
### 2. What the numbers say
The cascade design rests on two empirical facts.
#### 2.1 The sweep: bigger is not better
On a raw one-shot extraction task, twenty local models were compared. Key takeaways:
– **Bigger is not better:** A 4B model and a 7B model topped the leaderboard; a 12B and 14B sat below them. Parameter count alone poorly predicts accuracy, and the “next size up” is often not the next step in accuracy.
– **There is a floor:** Sub-2B models fell sharply in performance, and a 0.5B model often returned empty JSON. Starting too low wastes loop iterations on rungs that cannot hold.
– **Cost is not speed:** The hosted flagship was faster (about 1.4 seconds per field) than most local 7B–14B models (2.6–7.6 seconds). If latency matters, the cheapest local model is often the wrong default.
A small local model, by itself, achieved roughly one-third accuracy—usable only when backed by a tight retrieval context, strong prompts, validation, and code-driven formatting.
#### 2.2 The biggest lever is prompt content, not model size
Before scaling up models, scale the prompt. Adding the domain glossary, field synonyms, units, and definitions delivered the largest single accuracy jump:
– Small local model: 38% → 62% correct with glossary
– Flagship model: 62% → 100% correct with glossary
The same business vocabulary that boosts retrieval also fixes generation. Glossaries help every model tier, at a fraction of the cost of model upgrades.
—
### 3. The cascade: criteria, loop, and splitting
#### 3.1 Pick the starting model by criteria
The initial model is not chosen by size alone. Three criteria set the floor:
– **Confidentiality:** Confidential documents cannot leave the environment, forcing use of a local model.
– **Typed-output reliability:** The contract requires a strict JSON object with typed fields; weaker models drop or hallucinate fields.
– **Transformation complexity:** Hard one-shot transformations (e.g., reading “3M” and producing 3000000) trip small models even with good retrieval.
A simple dispatcher uses these criteria:
“`python
class GenerationPlan(BaseModel):
model: str
needs_validation: bool
needs_step_split: bool
max_escalations: int = 2
def plan_generation(doc, shape):
if doc.confidential:
return local_plan(shape)
return tier_plan(shape)
“`
#### 3.2 Validate, then escalate only on failure
Generation becomes a short, bounded loop:
“`python
def generate_with_escalation(question, context, plan, ladder):
for model in ladder.from_(plan.model, up_to=plan.max_escalations):
answer = generate(question, context, model=model)
if validate(answer, context):
return answer
return NotAnswered()
“`
A fast check (Article 8C) verifies JSON shape, citation spans, and source alignment. If validation passes, the cheap model’s work is shipped. If not, the call escalates to the next model up. The loop is bounded, so pathological fields eventually reach the strongest model and, if still wrong, return a typed “not answered.”
#### 3.3 Split the task, and the confidentiality cost you measure
When a model fails due to a hard transformation, decomposition can help:
– **Split approach:** Let the model point at the source line, and use code to normalize the value (e.g., “3M” → 3000000). This keeps documents local and offloads formatting to deterministic code.
– **Cost of splitting:** Accurate line pointing is itself hard; a wrong line dooms the field even if formatting is perfect. The benchmark shows that splitting can reduce accuracy compared to a capable model doing extraction and formatting in one pass.
Use splitting when confidentiality forces locality or when the failure mode is consistently a known transformation. Do not assume decomposition always wins; measure the trade-off.
—
### 4. Reconciling with question-side dispatch and validation
This cascade does not replace earlier design. Article 6C’s dispatcher proposes a model tier based on question type. Article 8C validates the answer. The pieces connect:
– **Question side:** Dispatcher suggests a starting tier.
– **Generation side:** The chosen model attempts extraction.
– **Validation side:** Article 8C confirms adequacy or triggers escalation.
Together they form a consistent, end-to-end loop: propose, attempt, judge, escalate.
—
### 5. Conclusion
Model choice is a brick-level decision, not a constant. The cheapest model is the right default when:
– A criteria-driven dispatcher selects a sensible starting rung,
– Validation guards every answer, and
– A bounded loop escalates only the cases that fail, splitting tasks when the failure is a transformation rather than model weakness.
The benchmark confirms that size does not order accuracy: small models on their own miss one-third of fields, while the glossary and prompt content move both small and flagship models toward perfection. Cost drops because most fields never leave the cheap rung; correctness holds because the hard cases get the stronger model, on evidence, not guesswork.
—
### 6. Further reading and sources
– Chen, Zaharia & Zou, *FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance* (arXiv 2305.05176, 2023)
– Madaan et al., *AutoMix: Automatically Mixing Language Models* (arXiv 2310.12963, 2023)
– Ong et al., *RouteLLM: Learning to Route LLMs with Preference Data* (arXiv 2406.18665, 2024)
– *Enterprise Document Intelligence* series: Article 6C (dispatch), Article 8C (validation), Article 10 (adaptive parsing)
The runnable notebook for this article is available at **doc-intel/notebooks-vol1**, reproducing the cascade, printing per-field cost, and showing escalation only where validation fails.
