**Rethinking RAG Generation: From Open-Ended Strings to Typed Contracts**
Modern Retrieval-Augmented Generation (RAG) systems often treat the Large Language Model (LLM) as an oracle, asking it to produce an answer string and trusting the result. However, when an answer is wrong, this approach offers no clarity on whether the error stems from retrieval, parsing, or the generation itself. This article, serving as a companion to the *Enterprise Document Intelligence* series, pushes back on that conventional pipeline and focuses on **brick 4 (generation)**. It argues that the key to reliable RAG is not better prompting for open-ended text, but a fundamental shift from strings to **typed, validated contracts**.
### The Problem with Treating the LLM as an Oracle
The mainstream RAG story is simple: retrieve relevant chunks, prompt the LLM with those chunks and a question, and receive an answer string. This flow has a critical flaw: it conflates *finding* information with *generating* an answer. When the result is incorrect, the term “hallucination” is often used as a catch-all explanation, which shuts down investigation. In a strict sense, hallucination refers to a model inventing facts from its parametric memory—ungrounded fabrication. In RAG, most wrong answers are not hallucinations in this pure sense but are instead caused by failures upstream in the extraction chain (poor parsing, flawed retrieval, or an inadequate generation contract).
Blaming “hallucination” ends the conversation. Naming the actual cause opens the door to a fix.
### The Solution: Generation as Filling a Typed Contract
The article proposes a paradigm shift. Instead of treating the answer as a raw string, the system should treat it as a **typed Pydantic object**. This object is defined by a clear schema that acts as a contract between the retrieval pipeline and the generation step.
* **The answer is not a string; it is a structured object.** For example, a question about a “premium amount” should return `Amount(value=124.0, currency=”USD”, unit=”month”, evidence=[…], answer_found=True, confidence=0.95)`, not the sentence “The premium is $124 per month.”
* **The LLM is a function, not an oracle.** Its role is to fill the schema fields, not to produce prose. A separate validator checks the output *before* it reaches the user.
* **The schema is the contract.** It dictates what data is required, how it should be formatted, and what self-assessment flags (like `answer_found` or `complete_answer_found`) should accompany it.
This approach provides several benefits: structured data for downstream consumption, clear citations linked to exact document spans, and explicit self-assessment fields that allow the system to flag uncertainty or partial answers.
### The Seven Generation Patterns for a Typed Contract
The core of the article outlines seven interconnected patterns that operationalize this typed-contract approach.
**Pattern 1: The LLM is a Function, Not an Oracle**
The generation brick’s sole purpose is to populate a predefined schema. This separates the extraction of information from the model’s parametric memory from the system’s understanding of what constitutes a valid answer.
**Pattern 2: Extract Typed Values, Never Compute**
The model should extract raw facts (e.g., `Amount(value=124, currency=”USD”)`), while Python code performs any necessary calculations (e.g., currency conversion) using a logged, auditable exchange rate. This preserves a clear audit trail.
**Pattern 3: Completeness from Structure, Not Self-Rating**
Instead of asking the LLM if it has retrieved all information, the system uses procedural checks. For example, if an answer spans a document section, the pipeline checks if the next page belongs to that same section. If it does, retrieval continues; if not, it stops. Completeness is a property of the retrieval process, not a guess made by the model.
**Pattern 4: Two Booleans, Not One Confidence Float**
A single floating-point confidence score conflates three distinct scenarios: a fully found answer, a partial answer, and an un-found answer. The pattern uses two separate boolean flags: `answer_found` and `complete_answer_found`. This allows the orchestration logic to make clear decisions: keep retrieving, ship the answer, or refuse.
**Pattern 5: One Prompt per Shape, Dispatched at Runtime**
Prompts should be composed from named fragments (e.g., a BASE prompt, a `Date` shape fragment, and a schema) rather than relying on a single, uncommented “mega-prompt.” This makes the system deterministic and traceable; if an answer is wrong six months later, engineers can reconstruct the exact prompt that was used from the logs.
**Pattern 6: No Reasoning Models on JSON Extraction**
If the schema already constrains the output to a specific format, adding a “reasoning” model provides no benefit but adds latency and cost. The model has no open-ended judgment to make; it simply extracts data. Use the smallest model capable of reliably filling the schema.
**Pattern 7: Decompose for Small Models, One Call for the Big Ones**
A frontier model can fill a complex, compound schema in a single call. A smaller model cannot and will silently invent values for fields it cannot deduce. The solution is to decompose the contract into stages. A small model makes a first call to extract raw data; Python then performs conversions and logic; a second, optional call can generate a final natural-language summary. The *schema* remains consistent; only the *granularity of calls* changes based on model capability.
### Across Sectors and Professions
This typed-contract discipline is universally applicable. Whether in legal, medical, financial, or technical domains, the validation logic remains the same. What changes is the **shape of the schema**. One sector might define a `MedicalDiagnosis` object with a `caveat` field for known unknowns, while another defines a `LegalClause` object that uses the two-boolean pattern to signal partial completeness. The validator, however, applies consistent checks for citation alignment, format validity, and boundary conditions across all sectors.
### FAQ
**Q: What is the difference between a “hallucination” and a failure in the RAG pipeline?**
A: In this framework, a “hallucination” is strictly a model invention with no grounding in the input data. Most RAG failures are not hallucinations but are caused by flaws in the upstream chain—retrieval of wrong chunks, faulty parsing of the document, or a poorly designed generation contract. The article argues for investigating this upstream chain rather than blaming the model.
**Q: What is the “naive baseline” that this article critiques?**
A: The naive baseline is the standard RAG pipeline: retrieve documents, prompt an LLM with “answer this question using the following context,” and receive a free-text answer. This frame has no mechanism to distinguish found from invented information, no way to record confidence, and no structured output for downstream systems to use.
**Q: What is a “typed Pydantic object” in this context?**
A: It is a structured data definition, typically using a library like Pydantic in Python. It acts as a contract that specifies the answer’s fields, their data types (e.g., `int`, `str`, `list`), and any metadata. For example, an answer object for a financial question would have fields for the numerical `value`, the `currency`, the `unit` (e.g., month), `evidence` spans pointing to the source text, and `answer_found` and `confidence` flags.
**Q: Why does the article argue against computation inside the LLM?**
A: When an LLM performs calculations (e.g., currency conversion), it does so invisibly. This erases the audit trail, making it impossible to verify the steps or replay the logic. The article’s “Extract, Compute, Verify” principle dictates that the LLM extracts the raw number, and Python code performs the calculation using a rate that is explicitly looked up and logged.
**Q: How does the “two booleans” pattern improve upon a single confidence score?**
A: A single confidence score (e.g., 0.6) is ambiguous. It could mean the answer is fully correct, half-correct, or entirely invented. Two booleans provide a clear trichotomy:
* `answer_found=True, complete_answer_found=True`: The answer is complete and can be shipped.
* `answer_found=True, complete_answer_found=False`: The question is answerable, but the retrieved information is partial, so the system should continue retrieving more documents.
* `answer_found=False`: The question is unanswerable from the current document, and the system should refuse to answer or escalate.
**Q: What is the “decomposition” pattern for small models?**
A: It is a strategy to handle the limitations of smaller models that cannot process a complex, multi-step schema in a single call. Instead of asking the small model to “extract, convert, and summarize” at once (which leads to invention), the contract is broken into discrete stages. For instance, one call extracts raw data, Python code performs logic (like a math conversion), and a second, simple call generates text. This ensures accuracy without sacrificing the benefits of a typed schema.
### Conclusion
Rethinking RAG generation is about moving from a passive, text-based paradigm to an active, engineering-centric one. By treating the LLM’s output as a typed contract rather than a string, teams gain auditability, verifiability, and robust error handling. The seven patterns presented provide a practical toolkit for implementing this shift. They demonstrate that a small, well-defined schema, combined with Python for computation and clear orchestration logic, can outperform a larger model operating in a “black box.” Ultimately, this approach transforms the LLM from an unreliable oracle into a reliable function within a larger, controllable system, ensuring that RAG deliverables are not just fluent, but fundamentally sound and trustworthy.
