**Article: Building RAG Pipelines That Actually Return Complete Lists**
Returning “five out of six” with full confidence is not a bug—it is a feature mismatch between retrieval and listing questions. Listing questions ask for *all* items, but standard top‑k RAG retrieves only a partial slice and then answers with whatever the LLM sees, even when critical items are absent. This article describes how to close that gap by changing retrieval shape, adding explicit completeness signals, and structuring answers so users can verify completeness.
—
### 1. Why Listing Breaks Naive RAG
Listing questions—such as “What are all the categories under the GOVERN function?” or “What are all the regularization techniques used to train the Transformer?”—require *every* item across the document. Standard RAG pipelines:
1. Embed the question.
2. Retrieve top‑k similar chunks.
3. Ask the LLM to extract the answer.
This works for factual questions (one passage, one answer) but fails for listing because:
– Items are scattered and never appear together.
– Items may be phrased differently (e.g., “Cybersecurity Supply Chain Risk Management,” “GV.SC,” “third‑party risk”).
– Top‑k truncates the set, and the LLM fills the silence with high‑confidence incompleteness.
Long‑context models hit the same wall: they surface likely candidates but still miss items expressed differently from the query.
—
### 2. Detecting Listing Questions Early
The pipeline must recognize a listing intent before retrieval. Signals include:
– Explicit markers: “list all,” “all categories,” “every technique.”
– Implicit plural intent: “what are all the…”
– Question classifiers or regex patterns that tag `intent: listing`.
Once tagged, orchestration routes the question to the listing branch (Article 13), which selects the appropriate retrieval strategy.
—
### 3. Three Strategies for Finding Every Item
#### 3.1 Strategy 1: Structural Retrieval (When It Works)
Leverage the document’s own enumeration:
– Table of Contents parent–child relationships (e.g., six children under “GV” for the NIST CSF).
– Section headings and bold subheadings (e.g., “Residual Dropout,” “Label Smoothing”).
– Numbered or bulleted lists.
Example: categories under GOVERN are immediate TOC children, returning six items in microseconds.
#### 3.2 Strategy 2: Pattern-Based Aggregation (Deterministic Sweep)
When items follow a recognizable pattern, sweep the document with regex:
– Subcategory codes matching `GV.[A-Z]{2}-d{2}`.
– Clause identifiers like `Section 5.x`.
– Entity patterns such as CVE IDs or model version tags.
A single pass can exhaustively return all matches, deduplicated by canonical name.
#### 3.3 Strategy 3: Semantic Aggregation with a Completeness Loop
For free‑form lists without structure or pattern:
– Broad retrieval (TOC‑driven or section‑level) feeds an LLM extraction.
– The LLM returns `items` plus `is_likely_complete` and optional `suggested_additional_keywords`.
– If completeness fails, expand keywords and retry within bounded iterations.
– Termination occurs on convergence, explicit cardinality confirmation, or iteration limit.
This loop is the listing analog of IRCoT-style retrieve‑then‑reason, scoped entirely within the listing branch.
—
### 4. From Items to a Shippable List
Finding items is only half the job; presentation must inspire trust.
#### 4.1 Deduplication
– Surface deduplication: exact string matches.
– Semantic deduplication: “Residual Dropout” vs. “dropout on residual connections” via LLM merging.
Each item should surface:
– Canonical name
– All surface forms
– Citations (page/line)
#### 4.2 Explicit Completeness Signals
Three sources, in precedence:
1. **Structural completeness**: guaranteed when retrieval is exhaustive (regex, TOC children).
2. **Cardinality cues**: document states “six Functions,” “three types of regularization.” Mismatch triggers iteration.
3. **LLM self‑assessment**: weak but useful when other signals are absent, bounded by max iterations.
The pipeline validates counts and revises retrieval when needed.
#### 4.3 Presentation Schema
A listing answer includes:
– Fully enumerated items with citations.
– Completeness verdict and source (structural, cardinality, or LLM assessment).
– Notes if items were missing or iterations occurred.
—
### 5. Two End‑to‑End Runs
#### 5.1 Counting Functions on the NIST CSF
– Intent: listing.
– TOC retrieval falls short; pattern sweep finds six function codes.
– Document states “six Functions.” Count matches → completeness verified.
– Output: clean, cited list with “complete” verdict.
#### 5.2 Regularization in the Transformer Paper
– Intent: listing.
– Section 5.4 retrieved; two items extracted.
– Document claims “three types of regularization.” Mismatch detected.
– LLM suggests “attention dropout.” Third item found on retry.
– Output: three cited items, completeness confirmed via iteration.
—
### 6. Conclusion
Listing questions expose a fundamental limitation in standard RAG: top‑k retrieval assumes one best passage, not an unknown number of scattered items. Effective listing pipelines must:
– Detect listing intent early.
– Choose retrieval strategy aligned with document structure (sweep over top‑k).
– Enforce completeness using document cardinality cues when available.
These principles ship as a dedicated listing module, composes cleanly with TOC navigation, cross‑references, and typed answer contracts. They turn “looks complete” into “is complete.”
—
### 7. Sources and Further Reading
– Amouyal et al. (QAMPARI, 2022): benchmark showing top‑k ceilings on list questions.
– Malaviya et al. (ExpertQA, NAACL 2024): per‑item attribution metrics.
– Min et al. (FActScore, EMNLP 2023): atomic‑fact decomposition.
– Asai et al. (Self‑RAG, ICLR 2024): reflection tokens; cardinality check as a stronger variant.
– Trivedi et al. (IRCoT, ACL 2023): retrieve‑then‑reason iteration.
Earlier in the series:
– From PDF to highlighted answer (four‑brick pipeline).
– Embeddings aren’t magic: predictable failure modes.
– RAG is not machine learning; route by question type.
– From regex to vision models: technique selection by document complexity.
Document parsing:
– Beyond extract_text: the two layers of a PDF.
– Stop returning flat text: the relational tables RAG needs.
Question parsing:
– Parsing user strings into retrieval briefs and generation briefs.
– What the question parser extracts.
– Dispatching parsed questions.
Retrieval and generation:
– Stop returning text: the answer contract.
– Assemble prompts with rules per question.
– Validating answers before the user sees them.
One‑document pipelines:
– Production RAG for PDFs end to end.
– Adaptive parsing and loop engineering in action.
– Cross‑reference loops.
