The Precisely as Designed. The Reply Was Nonetheless Unsuitable.
I need to let you know concerning the second I finished trusting retrieval scores.
I used to be operating a question towards a information base I had constructed fastidiously. Good chunking. Hybrid search. Reranking. The highest-k paperwork got here again with cosine similarities as excessive as 0.86. Each indicator stated the pipeline was working. I handed these paperwork to a QA mannequin, obtained a assured reply, and moved on.
The reply was incorrect.
Not hallucinated-wrong. Not retrieval-failed-wrong. The appropriate paperwork had come again. Each of them. A preliminary earnings determine and the audited revision that outdated it, sitting aspect by aspect in the identical context window. The mannequin learn each, selected one, and reported it with 80% confidence. It had no mechanism to inform me it had been requested to referee a dispute it was by no means designed to guage.
That’s the failure mode this text is about. It doesn’t present up in your retrieval metrics. It doesn’t set off your hallucination detectors. It lives within the hole between context meeting and technology — the one step within the RAG pipeline that nearly no one evaluates.
I constructed a reproducible experiment to isolate it. All the pieces on this article runs on a CPU in about 220 MB. No API key. No cloud. No GPU. The output you see within the terminal screenshots is unmodified.
Full Supply Code:
What the Experiment Assessments
The setup is intentionally medical. Three questions. One information base containing three conflicting doc pairs that make instantly contradictory claims about the identical reality. Retrieval is tuned to return each conflicting paperwork each time.
The query will not be whether or not retrieval works. It does. The query is: what does the mannequin do if you hand it a contradictory temporary and ask it to reply with confidence?
The reply, as you will note, is that it picks a aspect. Silently. Confidently. With out telling you it had a option to make.
Three Eventualities, Every Drawn from Manufacturing
Situation A — The restatement no one informed the mannequin about
An organization’s This autumn earnings launch stories annual income of $4.2M for fiscal yr 2023. Three months later, exterior auditors restate that determine to $6.8M. Each paperwork reside within the information base. Each are listed. When somebody asks “What was Acme Corp’s revenue for fiscal year 2023?” — each come again, with similarity scores of 0.863 and 0.820 respectively.
The mannequin solutions $4.2M.
It selected the preliminary determine over the audited revision as a result of the preliminary doc scored marginally greater in retrieval. Nothing concerning the reply alerts {that a} extra authoritative supply disagreed.
Situation B — The coverage replace that arrived too late
A June 2023 HR coverage mandates three days per week in-office. A November 2023 revision explicitly reverses it — absolutely distant is now permitted. Each paperwork are retrieved (similarity scores 0.806 and 0.776) when an worker asks concerning the present distant work coverage.
The mannequin solutions with the June coverage. The stricter, older rule. The one which now not applies.
Situation C — The API docs that by no means obtained deprecated
Model 1.2 of an API reference states a charge restrict of 100 requests per minute. Model 2.0, revealed after an infrastructure improve, raises it to 500. Each are retrieved (scores 0.788 and 0.732).
The mannequin solutions 100. A developer utilizing this reply to configure their charge limiter will throttle themselves to one-fifth of their precise allowance.
None of those are edge circumstances. Each manufacturing information base accumulates precisely these patterns over time: monetary restatements, coverage revisions, versioned documentation. The pipeline has no layer that detects or handles them.
Working the Experiment
pip set up -r necessities.txt
python rag_conflict_demo.pynecessities.txt
sentence-transformers>=2.7.0 # all-MiniLM-L6-v2 (~90 MB)
transformers>=4.40.0 # deepset/minilm-uncased-squad2 (~130 MB)
torch>=2.0.0 # CPU-only is okay
numpy>=1.24.0
colorama>=0.4.6Two fashions. One for embeddings, one for extractive QA. Each obtain routinely on first run and cache domestically. Whole: ~220 MB. No authentication required.
Section 1: What Naive RAG Does
Right here is the unmodified terminal output from Section 1 — commonplace RAG with no battle dealing with:
────────────────────────────────────────────────────────────────────
NAIVE | Situation A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal yr 2023?
Reply : $4.2M
Confidence : 80.3%
Battle : YES — see warning
Sources retrieved
[0.863] This autumn-2023-Earnings-Launch (2024-01-15)
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle pairs
fin-001 ↔ fin-002
numerical contradiction (topic_sim=0.83)
[Q4-2023-Earnings-Release: {'$4.2M'}] vs [2023-Annual-Report-Revised: {'$6.8M'}]
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | Situation B — Coverage Battle
────────────────────────────────────────────────────────────────────
Question : What's the present distant work coverage for workers?
Reply : all workers are required to be current within the workplace
a minimal of three days per week
Confidence : 78.3%
Battle : YES — see warning
Sources retrieved
[0.806] HR-Coverage-June-2023 (2023-06-01)
[0.776] HR-Coverage-November-2023 (2023-11-15)
[0.196] HR-Coverage-November-2023 (2023-11-15)
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | Situation C — Technical Battle
────────────────────────────────────────────────────────────────────
Question : What's the API charge restrict for the usual tier?
Reply : 100 requests per minute
Confidence : 81.0%
Battle : YES — see warning
Sources retrieved
[0.788] API-Reference-v1.2 (2023-02-10)
[0.732] API-Reference-v2.0 (2023-09-20)
[0.383] API-Reference-v2.0 (2023-09-20)
────────────────────────────────────────────────────────────────────
Three questions. Three incorrect solutions. Confidence between 78% and 81% on each one among them.
Discover what is going on within the logs earlier than every response:
09:02:20 | WARNING | Battle detected: {('fin-001', 'fin-002'): "numerical contradiction..."}
09:02:24 | WARNING | Battle detected: {('hr-001', 'hr-002'): "contradiction signal asymmetry..."}
09:02:25 | WARNING | Battle detected: {('api-001', 'api-002'): "contradiction signal asymmetry..."}The conflicts are detected. They’re logged. After which, as a result of resolve_conflicts=False, the pipeline passes the total contradictory context to the mannequin and solutions anyway. That warning goes nowhere. In a manufacturing system with no battle detection layer, you wouldn’t even get the warning.
Why the Mannequin Behaves This Method
This requires a second of clarification, as a result of the mannequin will not be damaged. It’s doing precisely what it was skilled to do.
deepset/minilm-uncased-squad2 is an extractive QA mannequin. It reads a context string and selects the span with the very best mixed start-logit and end-logit rating. It has no output class for “I see two contradictory claims.” When the context comprises each $4.2M and $6.8M, the mannequin computes token-level scores throughout your complete string and selects whichever span wins.
That choice is pushed by elements that don’t have anything to do with correctness [8]. The 2 major drivers are:
Place bias. Earlier spans within the context obtain marginally greater consideration scores as a result of encoder structure. The preliminary doc ranked greater in retrieval and due to this fact appeared first.
Language energy. Direct declarative statements (“revenue of $4.2M”) outscore hedged or conditional phrasing (“following restatement… is $6.8M”).
A 3rd contributing issue is lexical alignment — spans whose vocabulary overlaps extra intently with the query tokens rating greater no matter whether or not the underlying declare is present or authoritative.
Critically, what the mannequin does not think about in any respect: supply date, doc authority, audit standing, or whether or not one declare supersedes one other. These alerts are merely invisible to the extractive mannequin.
The identical dynamic performs out in generative LLMs, however much less visibly — the mannequin paraphrases relatively than extracting verbatim spans, so the incorrect reply is wearing fluent prose. The mechanism is similar. Joren et al. (2025) show at ICLR 2025 that frontier fashions together with Gemini 1.5 Professional, GPT-4o, and Claude 3.5 steadily produce incorrect solutions relatively than abstaining when retrieved context is inadequate to reply the question — and that this failure will not be mirrored within the mannequin’s expressed confidence.
The failure will not be a mannequin deficiency. It’s an architectural hole: the pipeline has no stage that detects contradictions earlier than handing context to technology.
Constructing the Battle Detection Layer
The detector sits between retrieval and technology. It examines each pair of retrieved paperwork and flags contradictions earlier than the QA mannequin sees the context. Crucially, embeddings for all retrieved paperwork are computed in a single batched ahead go earlier than pair comparability begins — every doc is encoded precisely as soon as, no matter what number of pairs it participates in.
Two heuristics do the work.
Heuristic 1: Numerical Contradiction
Two topic-similar paperwork that include non-overlapping significant numbers are flagged. The implementation filters out years (1900–2099) and naked small integers (1–9), which seem ubiquitously in enterprise textual content and would generate fixed false positives if handled as declare values.
@classmethod
def _extract_meaningful_numbers(cls, textual content: str) -> set[str]:
outcomes = set()
for m in cls._NUM_RE.finditer(textual content):
uncooked = m.group().strip()
numeric_core = re.sub(r"[$€£MBK%,]", "", uncooked, flags=re.IGNORECASE).strip()
strive:
val = float(numeric_core)
besides ValueError:
proceed
if 1900 <= val <= 2099 and "." not in numeric_core:
proceed # skip years
if val < 10 and re.fullmatch(r"d+", uncooked):
proceed # skip naked small integers
outcomes.add(uncooked)
return outcomesUtilized to Situation A: fin-001 yields {'$4.2M'}, fin-002 yields {'$6.8M'}. Empty intersection — battle detected.
Heuristic 2: Contradiction Sign Asymmetry
Two paperwork discussing the identical subject, the place one comprises contradiction tokens the opposite doesn’t, are flagged. The token set splits into two teams saved as separate frozenset objects:
_NEGATION_TOKENS: “not”, “never”, “no”, “cannot”, “doesn’t”, “isn’t”, and associated types_DIRECTIONAL_TOKENS: “increased”, “decreased”, “reduced”, “eliminated”, “removed”, “discontinued”
These are unioned into CONTRADICTION_SIGNALS. Retaining them separate makes domain-specific tuning simple — a authorized corpus may want a broader negation set; a changelog corpus may want extra directional tokens.
Utilized to Situation B: hr-002 comprises “no” (from “no longer required”); hr-001 doesn’t. Asymmetry detected. Utilized to Situation C: api-002 comprises “increased”; api-001 doesn’t. Asymmetry detected.
Each heuristics require topic_sim >= 0.68 earlier than firing. This threshold gates out unrelated paperwork that occur to share a quantity or a negation phrase. The 0.68 worth was calibrated for this doc set with all-MiniLM-L6-v2 — deal with it as a place to begin, not a common fixed. Totally different embedding fashions and totally different domains would require recalibration.
The Decision Technique: Cluster-Conscious Recency
When conflicts are detected, the pipeline resolves them by preserving probably the most just lately timestamped doc from every battle cluster. The important thing design resolution is cluster-aware.
A top-k consequence might include a number of impartial battle clusters — two monetary paperwork disagreeing on income and two API paperwork disagreeing on charge limits, all in the identical top-3 consequence. A naive method — preserve solely the only most up-to-date doc from the mixed conflicting set — would silently discard the profitable doc from each cluster besides probably the most just lately revealed one general.
As an alternative, the implementation builds a battle graph, finds linked elements by way of iterative DFS, and resolves every element independently:
@staticmethod
def _resolve_by_recency(
contexts: listing[RetrievedContext],
battle: ConflictReport,
) -> listing[RetrievedContext]:
# Construct adjacency listing
adj: dict[str, set[str]] = defaultdict(set)
for a_id, b_id in battle.conflict_pairs:
adj[a_id].add(b_id)
adj[b_id].add(a_id)
# Related elements by way of iterative DFS
visited: set[str] = set()
clusters: listing[set[str]] = []
for begin in adj:
if begin not in visited:
cluster: set[str] = set()
stack = [start]
whereas stack:
node = stack.pop()
if node not in visited:
visited.add(node)
cluster.add(node)
stack.lengthen(adj[node] - visited)
clusters.append(cluster)
all_conflicting_ids = set().union(*clusters) if clusters else set()
non_conflicting = [c for c in contexts if c.document.doc_id not in all_conflicting_ids]
resolved_docs = []
for cluster in clusters:
cluster_ctxs = [c for c in contexts if c.document.doc_id in cluster]
# ISO-8601 timestamps type lexicographically — max() provides most up-to-date
finest = max(cluster_ctxs, key=lambda c: c.doc.timestamp)
resolved_docs.append(finest)
return non_conflicting + resolved_docsNon-conflicting paperwork go by way of unchanged. Every battle cluster contributes precisely one winner.
Section 2: What Battle-Conscious RAG Does
────────────────────────────────────────────────────────────────────
RESOLVED | Situation A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal yr 2023?
Reply : $6.8M
Confidence : 79.6%
Battle : RESOLVED
⚠ Conflicting sources detected — reply derived from most up-to-date
doc per battle cluster.
Sources retrieved
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle cluster resolved: saved '2023-Annual-Report-Revised' (2024-04-03),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | Situation B — Coverage Battle
────────────────────────────────────────────────────────────────────
Reply : workers are now not required to keep up
a set in-office schedule
Confidence : 78.0%
Battle : RESOLVED
Battle cluster resolved: saved 'HR-Coverage-November-2023' (2023-11-15),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | Situation C — Technical Battle
────────────────────────────────────────────────────────────────────
Reply : 500 requests per minute
Confidence : 80.9%
Battle : RESOLVED
Battle cluster resolved: saved 'API-Reference-v2.0' (2023-09-20),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
Three questions. Three right solutions. The boldness scores are virtually an identical to Section 1 — 78–81% — which underscores the unique level: confidence was by no means the sign that one thing had gone incorrect. It nonetheless will not be. The one factor that modified is the structure.
What the Heuristics Can not Catch
I need to be exact concerning the failure envelope, as a result of a way that understates its personal limitations will not be helpful.
Paraphrased conflicts. The heuristics catch numerical variations and express contradiction tokens. They won’t catch “the service was retired” versus “the service is currently available.” That may be a actual battle with no numeric distinction and no negation token. For these, a Pure Language Inference mannequin — cross-encoder/nli-deberta-v3-small at ~80 MB — can rating entailment versus contradiction between sentence pairs. That is the extra sturdy path described within the educational literature (Asai et al., 2023), and the ConflictDetector class is designed to be prolonged on the _pair_conflict_reason methodology for precisely this function.
Non-temporal conflicts. Recency-based decision is acceptable for versioned paperwork and coverage updates. It isn’t acceptable for knowledgeable opinion disagreements (the minority view could also be right), cross-methodology knowledge conflicts (recency is irrelevant), or multi-perspective queries (the place surfacing each views is the correct response). In these circumstances, the ConflictReport knowledge construction gives the uncooked materials to construct a distinct response — surfacing each claims, flagging for human evaluation, or asking the consumer for clarification.
Scale. Pair comparability is O(k²) in retrieved paperwork. For okay=3 that is trivial; for okay=20 it’s nonetheless superb. For pipelines retrieving okay=100 or extra, pre-indexing identified battle pairs or cluster-based detection turns into needed.
The place the Analysis Neighborhood Is Taking This
What you have got seen here’s a sensible heuristic approximation of an issue that energetic analysis is attacking at a way more refined stage.
Cattan et al. (2025) launched the CONFLICTS benchmark — the primary particularly designed to trace how fashions deal with information conflicts in reasonable RAG settings. Their taxonomy identifies 4 battle classes — freshness, conflicting opinions, complementary info, and misinformation — every requiring distinct mannequin behaviour. Their experiments present that LLMs steadily fail to resolve conflicts appropriately throughout all classes, and that explicitly prompting fashions to purpose about potential conflicts considerably improves response high quality, although substantial room for enchancment stays.
Ye et al. (2026) launched TCR (Clear Battle Decision), a plug-and-play framework that disentangles semantic relevance from factual consistency by way of twin contrastive encoders. Self-answerability estimation gauges confidence within the mannequin’s parametric reminiscence, and the ensuing scalar alerts are injected into the generator by way of light-weight soft-prompt tuning. Throughout seven benchmarks, TCR improves battle detection by 5–18 F1 factors whereas including solely 0.3% parameters.
Gao et al. (2025) launched CLEAR (Battle-Localized and Enhanced Consideration for RAG), which probes LLM hidden states on the sentence illustration stage to detect the place conflicting information manifests internally. Their evaluation reveals that information integration happens hierarchically and that conflicting versus aligned information displays distinct distributional patterns inside sentence-level representations. CLEAR makes use of these alerts for conflict-aware fine-tuning that guides the mannequin towards correct proof integration.
The constant discovering throughout all of this work matches what this experiment demonstrates instantly: retrieval high quality and reply high quality are distinct dimensions, and the hole between them is bigger than the group has traditionally acknowledged.
The distinction between that analysis and this text is 220 MB and no authentication.
What You Ought to Truly Do With This
1. Add a battle detection layer earlier than technology. The ConflictDetector class is designed to drop into an present pipeline on the level the place you assemble your context string. Even the 2 easy heuristics right here will catch the patterns that seem most frequently in enterprise corpora: restatements, coverage updates, versioned documentation.
2. Distinguish battle sorts earlier than resolving. A temporal battle (use the newer doc) is a distinct downside from a factual dispute (flag for human evaluation) or an opinion battle (floor each views). A single decision technique utilized blindly creates new failure modes.
3. Log each ConflictReport. After per week of manufacturing visitors you’ll understand how usually your particular corpus generates conflicting retrieved units, which doc pairs battle most steadily, and what question patterns set off conflicts. That knowledge is extra actionable than any artificial benchmark.
4. Floor uncertainty if you can not resolve it. The appropriate reply to an unresolvable battle is to not choose one and conceal the selection. The warning discipline in RAGResponse is there exactly to help responses like: “I found conflicting information on this topic. The June 2023 policy states X; the November 2023 update states Y. The November document is more recent.”
Working the Full Demo
# Full output with INFO logs
python rag_conflict_demo.py
# Demo output solely (suppress mannequin loading logs)
python rag_conflict_demo.py --quiet
# Run unit assessments with out downloading fashions
python rag_conflict_demo.py --test
# Plain terminal output for log seize / CI
python rag_conflict_demo.py --no-colorAll output proven on this article is unmodified output from an area Home windows machine operating Python 3.9+ in a digital setting. The code and output are absolutely reproducible by any reader with the listed dependencies put in.
The Takeaway
The retrieval downside is essentially solved. Vector search is quick, correct, and well-understood. The group has spent years optimising it.
The context-assembly downside will not be solved. No person is measuring it. The hole between “correct documents retrieved” and “correct answer produced” is actual, it’s common, and it produces assured incorrect solutions with no sign that something went incorrect.
The repair doesn’t require a bigger mannequin, a brand new structure, or further coaching. It requires one further pipeline stage, operating on embeddings you have already got, at zero marginal latency.
The experiment above runs in about thirty seconds on a laptop computer. The query is whether or not your manufacturing system has the equal layer — and if not, what it’s silently answering incorrect proper now.
References
[1] Ye, H., Chen, S., Zhong, Z., Xiao, C., Zhang, H., Wu, Y., & Shen, F. (2026). Seeing by way of the battle: Clear information battle dealing with in retrieval-augmented technology. arXiv:2601.06842.
[2] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Studying to retrieve, generate, and critique by way of self-reflection. arXiv:2310.11511.
[3] Cattan, A., Jacovi, A., Ram, O., Herzig, J., Aharoni, R., Goldshtein, S., Ofek, E., Szpektor, I., & Caciularu, A. (2025). DRAGged into conflicts: Detecting and addressing conflicting sources in search-augmented LLMs. arXiv:2506.08500.
[4] Gao, L., Bi, B., Yuan, Z., Wang, L., Chen, Z., Wei, Z., Liu, S., Zhang, Q., & Su, J. (2025). Probing latent information battle for trustworthy retrieval-augmented technology. arXiv:2510.12460.
[5] Jin, Z., Cao, P., Chen, Y., Liu, Ok., Jiang, X., Xu, J., Li, Q., & Zhao, J. (2024). Tug-of-war between information: Exploring and resolving information conflicts in retrieval-augmented language fashions. arXiv:2402.14409.
[6] Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2025). Ample context: A brand new lens on retrieval augmented technology programs. arXiv:2411.06037.
[7] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … & Kiela, D. (2020). Retrieval-augmented technology for knowledge-intensive NLP duties. arXiv:2005.11401.
[8] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023). When to not belief language fashions: Investigating effectiveness of parametric and non-parametric reminiscences. arXiv:2212.10511.
[9] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings utilizing Siamese BERT-networks. arXiv:1908.10084.
[10] Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Data conflicts for LLMs: A survey. arXiv:2403.08319.
[11] Xie, J., Zhang, Ok., Chen, J., Lou, R., & Su, Y. (2023). Adaptive chameleon or cussed sloth: Revealing the conduct of enormous language fashions in information conflicts. arXiv:2305.13300.
Full Supply Code:
Fashions Used
Each fashions obtain routinely on first run and cache domestically. No API key or HuggingFace authentication is required.
Disclosure
All code was written, debugged, and validated by the creator by way of a number of iterations of actual execution. All terminal output on this article is unmodified output from an area Home windows machine operating Python 3.9+ in a digital setting. The code and output are absolutely reproducible by any reader with the listed dependencies put in.
The creator has no monetary relationship with Hugging Face, deepset, or any organisation referenced on this article. Mannequin and library selections have been made solely on the premise of measurement, licence, and CPU compatibility.
