**Optimizing Token Costs: Sequential vs. Batch Feeding in Enterprise RAG**
Modern Retrieval-Augmented Generation (RAG) pipelines typically adopt a fixed, batch approach where the language model receives all retrieved candidates (top‑K) in a single call. While simple, this strategy is inefficient for the majority of straightforward, factual queries where the first retrieved chunk already contains the complete answer. This article examines a sequential alternative—top‑1 first with a sufficiency check—and explains when each strategy should be used, how to decide per question, and the tangible cost savings that result.
### Two Regimes for Feeding Top‑K to Generation
Retrieval hands the generation step an ordered list of candidates. There are two primary ways to consume these candidates:
#### 1.1 The Fixed Top‑K Pipeline and What It Wastes
The default pattern in many RAG tutorials is straightforward:
“`python
top_k = retrieval(question, k=5)
answer = generation(question, top_k)
“`
This pipeline runs retrieval once, then passes all K chunks to the LLM in a single call. The token cost is approximately `generation_cost(question + 5 × chunk_size)`. Even when the first chunk fully and correctly answers the question, the remaining chunks add no value but still incur token costs.
For example, a user asking *“what is the effective date of this policy?”* will trigger retrieval of several line‑windows containing “effective.” The first window yields the correct date; the rest are noise. In a single query this overhead is negligible, but across tens of thousands of documents it represents real, avoidable spend.
#### 1.2 Sequential: Top‑1 First, Sufficiency Predicate, Escalate If Needed
The sequential regime treats the candidate list as an ordered queue and evaluates sufficiency at each step:
“`python
for i, candidate in enumerate(top_k):
answer = generation(question, [candidate])
if answer.answer_found and answer.complete_answer_found:
break
“`
This relies on the typed contract introduced in Article 8A, where the generation returns:
– `answer_found`: whether the candidate contains the needed information
– `complete_answer_found`: whether the candidate contains the full answer, not just a fragment
If both flags are true, the loop stops. For the effective‑date example, the LLM evaluates the first candidate, sets both flags to true, and exits after consuming just one chunk. Token cost drops to roughly `generation_cost(question + 1 × chunk_size)`—a potential 80% reduction compared to batch for straightforward queries.
If the top‑1 is insufficient, the loop proceeds to the next candidate, continuing until sufficiency is met or the K candidates are exhausted.
#### 1.3 Batch: Send All K at Once, Let the LLM Arbitrate
Batch mode retains the traditional single‑call behavior. It remains the right choice for three classes of questions:
– **Listing**: e.g., “list all exclusions in this contract.” The answer requires every matching item, so stopping at the first correct chunk would lose information.
– **Comparison**: e.g., “is the premium higher than last year’s?” Both current and prior values must be considered together.
– **Tight‑score retrieval**: When top‑K candidates have similar relevance scores, the LLM needs full context to arbitrate confidently.
Cost-wise, batch always pays the higher price of processing K chunks in one call. Sequential pays that higher price only for the minority of complex or multi‑fact queries, while saving heavily on the majority of factual lookups. On a typical enterprise workload (K = 5, 80% of questions resolved by top‑1), sequential can reduce generation token consumption by around 65%.
### The Dispatch Decision: Per Question, Not Per Pipeline
Rather than committing to one strategy globally, the architecture should route each question based on its shape. The question parser, enriched in Article 2, sets two key attributes:
– `question_df.answer_shape`
– `question_df.decomposition`
A deterministic dispatcher uses these attributes to select sequential or batch mode:
– **Sequential** for single‑value extractions (date, amount, boolean)
– **Batch** for lists, comparisons, and tightly scored sets
This decision logic is consistent across domains—legal, insurance, medical, financial, and compliance—because the underlying question patterns recur everywhere. Unlike an LLM‑driven agent, the deterministic dispatcher guarantees identical questions follow identical paths, which is essential for auditability and reproducibility.
### The Sufficiency Signal
Sequential mode depends on a reliable self‑assessment from the generation step. The typed contract from Article 8A provides two boolean signals:
– `answer_found`
– `complete_answer_found`
These flags replace unreliable confidence scores and heuristic thresholds. The loop follows a simple policy:
– `found + complete` → stop and return the answer
– `found + incomplete` → continue to the next candidate
– `not found` → continue or give up after K attempts
An additional budget guard prevents runaway loops, ensuring even sequential execution respects token or time limits.
### Cost, and Where the Series Stops
#### A Concrete Cost Comparison
Consider a daily workload of 100 insurance questions, K = 5, average chunk size = 600 tokens:
– **Batch**: 100 × generation cost over 5 chunks ≈ 330k input tokens per day
– **Sequential** (80% top‑1 sufficient): ≈ 115k input tokens per day
This translates to roughly 65% saving in generation tokens for the majority of queries, while preserving correctness for edge cases.
#### The Boundary of the Series
The article deliberately avoids an LLM‑based dispatcher that would choose batch or sequential per question. Such an “agentic” approach would compromise determinism and auditability. The deterministic dispatcher described here—driven by parsed question structure rather than runtime model opinion—ensures consistent behavior, which is foundational for compliance, debugging, and cost predictability.
### FAQ
**Q: What is the difference between batch and sequential feeding of top‑K?**
A: Batch sends all K retrieved chunks to the LLM in one call. Sequential starts with the top‑1 chunk and, only if the answer is incomplete, proceeds down the list. Sequential reduces token usage when the first chunk is sufficient, while batch is necessary for questions requiring all candidates at once.
**Q: How does the system decide which strategy to use?**
A: The question parser determines `answer_shape` and `decomposition`. A deterministic rules dispatcher routes factual, single‑value questions to sequential mode and list/comparison questions to batch mode.
**Q: What role does the typed contract play?**
A: The typed contract, specifically `AnswerWithEvidence`, supplies boolean flags (`answer_found`, `complete_answer_found`) that let the sequential loop know when to stop. This removes the need for confidence thresholds and makes the decision robust and model‑stable.
**Q: Are there any edge cases where sequential can miss the answer?**
A: If the correct information appears only beyond the first candidate and the top‑1 is flagged as not found or incomplete, the loop will continue. The bounded loop ensures all K candidates are considered if needed, so no information is lost.
**Q: Does sequential mode increase latency?**
A: In the common case where top‑1 suffices, latency is lower because the loop stops after one LLM call. In worst‑case scenarios (e.g., listing), latency matches batch since all candidates are evaluated, but correctness is preserved.
**Q: Is an LLM‑driven dispatcher ever appropriate?**
A: An agentic dispatcher could dynamically learn routing patterns, but it would sacrifice determinism and auditability. This article adopts a deterministic dispatcher to ensure consistent behavior and traceability, especially important in regulated domains.
### Conclusion
Blindly feeding all retrieval candidates to the language model is simple but wasteful for the majority of factual enterprise queries. By introducing a sufficiency predicate and processing candidates sequentially—top‑1 first with early stopping—organizations can achieve significant token savings without sacrificing correctness. The key insight is that routing should be determined by question shape, not by the language model. A deterministic dispatcher, powered by typed generation signals, provides an auditable, efficient, and domain‑agnostic mechanism to choose between sequential and batch feeding. For most enterprise workloads, this hybrid approach delivers the best balance of cost, speed, and reliability.
