# Why RAG Systems Fail in Production — And How to Test for It
## Introduction
Retrieval-augmented generation (RAG) is one of the most widely adopted patterns for building reliable question-answering systems. The core idea is straightforward: take a user’s question, search through a collection of documents for relevant passages, and feed those passages to a language model so it can generate an accurate answer. This document collection — the body of text the system searches through — is often referred to as the corpus.
In a controlled setting, evaluating a RAG pipeline is relatively simple. You submit a well-crafted question, the retriever pulls the right passage from a clean corpus, and the language model produces a correct answer. Everything works.
But real-world document collections are never clean. They drift over time, accumulate errors, and contain inconsistencies that no one anticipated. These imperfections can silently break your RAG system, sending the retriever to the wrong passage and causing even the best language model to generate confidently wrong answers.
This article walks through four common problems that corrupt production document collections, demonstrates how each one breaks a retrieval pipeline, and shows practical fixes you can apply to harden your system against failure.
—
## The Toy Retrieval Pipeline
Before diving into the problems, it helps to understand the basic mechanics at play. A retriever works by breaking documents and queries into small units of text called chunks, indexing them, and then scoring how well each document matches a given question. In our simplified example, we use four short documents and a scoring function that counts how many query words appear in each document. The document with the highest count wins.
Each document in our pipeline carries four fields: an identifier, the text content, a topic label, and a last-updated date. While production systems typically use far more sophisticated matching methods — embedding-based similarity, hybrid search, re-ranking — the same types of input problems that break this simple setup will also affect more complex ones. The toy pipeline just makes each failure easier to spot.
—
## Problem 1: Stale Documents That Contradict Each Other
Document collections evolve. A returns policy might change from a 60-day window to a 30-day window, but the ingestion process often copies the new version into the index without removing the old one. Now both versions exist and are searchable.
When a user asks, “How many days is the returns window?”, both documents match equally well — each contains the words “returns” and “days.” The retriever faces a tie and, in our example, defaults to list order, which means it returns the older 2024 policy. A customer could be told they have 60 days to return an item when the actual limit is 30.
The fix here is to introduce a recency tiebreaker: when documents receive the same score, prefer the one with the more recent updated date. This works well when newer documents clearly replace older ones. In other collections, you may need an explicit status field — such as “current” or “retired” — especially when a newer document coexists with an older one rather than replacing it outright.
## Problem 2: OCR Errors That Hide Matching Words
Many production document collections include scanned PDFs. Optical character recognition (OCR) software extracts text from these scans, but it frequently makes subtle substitution errors. A capital letter “I” can be misread as the number “1,” and the letter “O” can become the number “0.” These errors are invisible to a human skimming a page but devastating to a token-based retriever.
If the original document says “Enterprise plan includes SSO for employees,” a faulty OCR output might read “Enterpr1se” and “SS0.” When a user queries “Does Enterprise include SSO?”, the retriever looks for exact token matches. Since the document now contains “enterpr1se” and “ss0” while the query contains “enterprise” and “sso,” neither word matches. Every document scores zero, and the retriever falls back to list order — returning the wrong document entirely.
The correction is a normalization function that corrects known OCR substitutions before tokenization. Applying it consistently to both queries and documents ensures the scorer sees clean, comparable text. However, care is needed: converting every “0” to “o” could corrupt product codes, serial numbers, or measurements. Build your substitution rules from actual OCR errors found in your own extracted documents, and restrict them to fields where the change is safe.
## Problem 3: A Typo in the Query
End users are not careful typists. Someone searching for “warehuse sync” has typed one letter too few — “warehouse” is misspelled. An exact token matcher treats “warehuse” as a completely unrelated word, contributing zero points to the score.
In our example, the misspelled word leaves only “sync” to determine the result. Both the CRM document and the warehouse inventory document contain this word, so another tie occurs — this time favoring the CRM document because it appears earlier in the list. The user gets information about contact sync instead of inventory sync.
Fuzzy matching solves this by comparing words based on spelling similarity and giving close matches partial credit. When fuzzy matching is enabled, “warehuse” is close enough to “warehouse” for the inventory document to score higher and rank first. The key parameter here is the similarity threshold: set it too aggressively and unrelated words start matching; set it too conservatively and common typos still fail. The best thresholds come from testing against real user queries, not a handful of invented spelling mistakes.
## Problem 4: A Table Split Across a Page Boundary
Some PDF extraction tools create one chunk per page. When a table spans multiple pages, this creates a serious problem: the first chunk might contain a row label like “Basic | Storage” while the second chunk contains only the value, “10 GB.”
In our example, the first page-chunk contains all three query tokens — “basic,” “plan,” and “storage” — so it scores highest. But its text ends abruptly after the label. The value, “10 GB,” lives in the next chunk and may never reach the language model. The retriever returns a fragment that mentions storage but provides no actual number, leaving the language model without the evidence it needs.
The fix belongs in the ingestion process, not the retrieval logic. When the pipeline detects table fragments, it should join related pages before creating searchable chunks. The key is to be surgical: joining every pair of pages would create oversized chunks that mix unrelated text. Instead, limit the rule to detected tables or carry enough neighboring content forward to preserve each complete row.
—
## Testing the Evidence Inside Each Result
Returning the correct document ID is only half the battle. An evidence check confirms whether the retrieved chunk actually contains the information needed to answer the question.
Consider the divided table problem again: a chunk from the correct limits document might contain the words “Basic” and “Storage” while leaving “10 GB” on the next page. The document ID is right, but the evidence is incomplete. Without checking for the presence of required values, you might ship a system that consistently returns a document but still answers questions incorrectly.
A practical testing approach names the specific words or values that must appear in the retrieved text. These assertions serve a dual purpose: they catch retrieval failures, and they make those failures easy to diagnose. A wrong document ID points to a ranking or filtering issue. A correct ID with missing text points to extraction or chunking problems. A correct ID with all required evidence present means the retrieval step succeeded and any remaining errors belong to the generation step.
If your retriever returns multiple chunks, apply the same check to the combined text that gets passed to the language model. The evaluated text should match exactly what the model receives, giving you a true picture of retrieval quality.
—
## What the Four Tests Reveal
When these four corrupted inputs are run against an unprotected retriever, all four fail. After applying the matching correction for each scenario, all four return the expected document. The fixes are small and targeted:
– Use document dates to resolve score ties.
– Normalize known OCR character substitutions before matching.
– Allow close spelling matches for user queries.
– Preserve complete table rows during the ingestion process.
Each fix addresses a different root cause, and the separate test results tell you exactly which protection is missing. These four cases form a starting test suite. Run them alongside a standard relevance evaluation — which measures whether retrieval works on expected inputs — and you get fault-injection tests, which measure whether retrieval still works after a realistic defect has been introduced.
When production returns the wrong document, add a regression test: a repeatable check that confirms the bug stays fixed after future code changes. Over time, this growing suite becomes a safety net that catches regressions before they reach users.
—
## Frequently Asked Questions
**Q: Why not just use a more advanced retriever to avoid these problems?**
A: More advanced retrievers — embedding-based models, hybrid search, cross-encoders — handle many of these issues better than simple token matching, but they are not immune. Stale documents still contradict each other, OCR errors still degrade embedding quality, and divided tables still lose critical data during chunking. The principles of fault-injection testing apply regardless of retrieval sophistication.
**Q: How do I know which OCR substitutions are safe to normalize?**
A: Analyze a sample of your actual extracted documents and catalog the recurring errors. Then evaluate each substitution for side effects: would converting “0” to “o” turn a valid product code into an incorrect one? If so, exclude that field from normalization. Build rules from your own data, not generic heuristics.
**Q: What if the fuzzy matching threshold causes false positives?**
A: This is the core trade-off. A low threshold catches more typos but risks matching unrelated words. A high threshold avoids false positives but misses legitimate misspellings. The best way to find the right balance is to test against a representative sample of real user queries with known correct answers, adjusting the threshold until the false-positive and false-negative rates are acceptable.
**Q: How many fault-injection test cases do I need?**
A: Four is a starting point, not a finish line. Every time a real production failure occurs, add a regression test that reproduces it. The goal is coverage of the failure modes specific to your document collection and user population. Over time, your test suite becomes a map of the system’s known weaknesses.
**Q: Can evidence checking replace relevance evaluation?**
A: No. Evidence checking verifies that the retrieved chunk contains the right information. Relevance evaluation verifies that the right chunk was retrieved in the first place. They are complementary: a system can pass one and fail the other. Both should be part of your test suite.
**Q: What about joined tables creating oversized chunks?**
A: This is a real trade-off. Joining pages increases chunk size, which can push you past token limits for the language model or dilute relevance scores with unrelated text. The solution is to be targeted — detect table structures explicitly and join only the rows that belong together, rather than joining every consecutive page pair.
—
## Conclusion
RAG systems are powerful, but their reliability depends entirely on the quality of the data they search and the robustness of their retrieval logic. Production document collections are messy by nature — they contain outdated policies, OCR artifacts, user typos, and formatting edge cases that no clean evaluation would ever reveal.
The approach outlined here — building a simple retriever, deliberately introducing realistic defects, and writing tests that verify both the correct document and the correct evidence — gives you a systematic way to find and fix these weaknesses before they affect real users. The four fault-injection scenarios covered here are just the beginning. As your document collection grows and your user base diversifies, new failure modes will emerge. A disciplined testing practice ensures you are always ready for them.
Thank you for reading
