**Reimagining Enterprise Retrieval: From Embedding Reflex to Structured Filtering**
Most tutorials frame retrieval as “embed the question, find the nearest chunks, and rerank.” In the Enterprise Document Intelligence series, this default story is deliberately rejected. Instead, the third “brick” of our four‑brick architecture argues that retrieval in enterprise documents is best understood as **filtering on structured tables**, not free‑text search. Embeddings are an optional fallback, anchor and context are handled separately, and the whole flow can be made auditable and deterministic. The companion notebooks (doc‑intel/notebooks‑vol1) illustrate these ideas on real documents.
A single diagram captures the contrast: rather than one opaque cosine signal over chunked text, we run three signals in parallel—keyword on lines, TOC reasoning, and embedding cosine—and let an LLM arbitrate once at the end. This architecture yields measurable gains in precision, coverage, and debuggability. Below are the six core lessons the series emphasizes, each with concrete consequences for cost and reliability.
Lesson 1: Retrieval is filtering, not searching
Once parsing is complete, retrieval becomes a SQL‑like join between the question’s columns and the document’s columns. Filter keeps only rows that satisfy a boolean condition and returns zero when nothing matches; search scores all candidates and always returns something, even when the answer is absent. Because filters are inspectable code, they provide an audit trail that embeddings cannot match. For example, asking about “positional encoding” in the Attention paper can be answered by filtering line_df for that exact phrase and toc_df for the matching section title, with no cosine computation required.
→ See Article 7A for the full mental model.
Lesson 2: Anchor and context, kept apart
Retrieval should return both a precise anchor (the line containing the answer) and the appropriate surrounding scope (the context needed for generation). Top‑k chunking forces a compromise: small chunks lose context, large chunks lose precision. By treating keyword location and scope separately, we keep precision and coverage in one pass. The keyword finds the anchor; the section boundaries define what context travels to the generator.
→ See Article 7A for how the line between anchor and context is drawn.
Lesson 3: Embeddings come last, not first
Keywords and the document’s own TOC should be the primary signals; embeddings are the optional final step, used only when vocabulary mismatch is expected. On straightforward factual questions—such as “effective date?”—a cheap regex or keyword pass often suffices, avoiding cosine similarity entirely. Embeddings are therefore a fallback, not a foundation, reducing cost and latency where possible.
→ See Article 7B for building a three‑signal pipeline.
Lesson 4: Keywords prove absence; embeddings cannot
A zero keyword match means the answer is genuinely not in the document. A low embedding similarity, however, could mean mismatch in wording, domain phrasing, or synonymy. Consequently, keywords decide whether an answer exists, while embeddings refine relevance. This asymmetry makes keyword‑first retrieval the safer default for enterprise risk and compliance.
→ See Article 7B for the keyword‑first discipline explained.
Lesson 5: Co‑occurrence beats BM25 on narrow corpora
In small, known corpora, BM25’s IDF assumptions break down. Instead of counting term frequency, retrieval should favor the answer’s typical shape: a keyword next to a specific value (e.g., “deductible is $1000”). Co‑occurrence signals, combined with high‑value regex anchors, outperform statistical term weights when documents are limited and nearly every term is “rare.”
→ See Article 7B for comparing co‑occurrence against BM25.
Lesson 6: One LLM pass over the TOC
The table of contents (toc_df), typically 20–100 rows, can be handed to a small model to identify relevant sections. This single cached call catches paraphrases that keyword matching misses (e.g., “exit early” versus “Termination and Cancellation”) and acts as a powerful, deterministic retrieval signal.
→ See Article 7B for TOC reasoning and Article 7C for the LLM arbiter.
Across sectors and professions, the same three‑signal pattern holds: keyword on line_df, TOC reasoning on toc_df, and embedding cosine as a fallback. Embeddings may fire only when vocabulary diverges (for example, “tachycardia” versus “rapid heart rate”), while the majority of queries resolve with fast, deterministic signals. Keywords prove absence, the TOC provides structure, and anchor/scope separation preserves both precision and context. The six deep‑dives (7A, 7B, 7C, and 7bis) provide runnable code on real documents; this article is the catalogue that points to them.
**Source:** This article is adapted from “Brick 3 (retrieval) — Retrieval,” *Enterprise Document Intelligence* companion series, which lays out the philosophy of Amplify the Expert. The series’ companion notebooks and deeper explorations are available at doc-intel/notebooks-vol1.
