## RAG vs. Long Context: A Practical Experiment with Large Language Models
With context windows expanding to millions of tokens, a pressing question arises: Is Retrieval-Augmented Generation (RAG) still necessary? This article presents a controlled experiment comparing RAG to a “long context” approach—feeding an entire knowledge base into a model like Kimi K3—and explores the practical trade-offs beyond the theory.
### The Core Setup: One Corpus, Two Paths
The experiment used a corpus of 32 personal articles, totaling 127,068 tokens, all from the author. This size fits within the Kimi K3’s one-million-token window. The goal was to answer 12 identical questions using two distinct methods:
1. **The RAG Path:** The corpus was processed into 788 small chunks. For each question, the system would retrieve the top 5 most relevant chunks (about 1,200 tokens) and pass them to the model along with the question.
2. **The Long Context Path:** The entire 127,000+ token corpus was included in every single prompt, alongside the question.
Both paths used the same system instructions and the same model (Kimi K3 at `temperature=1`) to ensure a fair comparison. A blind evaluation was conducted, where the author graded the answers without knowing which method produced them, based on correctness, completeness, and whether answers were “grounded” in the provided text.
### The Questions: A Difficulty Gradient
The 12 questions were designed to test three different scenarios:
* **Group A (Single Fact):** Questions with a single, definitive answer found in one place (e.g., “Which embedding model was used in the chunk size experiment?”). This is the ideal use case for retrieval.
* **Group B (Cross-Article):** Questions requiring the synthesis of information from two or three specific articles (e.g., comparing advice on RAG and fine-tuning).
* **Group C (Corpus-Wide):** Questions that could only be answered by considering all 32 articles simultaneously (e.g., “How many articles link to a specific GitHub repository, and what are they?”).
### Critical Insights: What Went Wrong
The experiment uncovered significant pitfalls that are often overlooked when considering long-context models:
1. **The Empty Answer Bug:** In the first run, half of the long-context answers were completely empty. The cause was shocking: **the model’s “thinking” tokens consumed its entire output budget.** Reasoning models like Kimi K3 use tokens for internal thought processes before generating an answer. If the `max_completion_tokens` limit is reached, the model will “finish” with a `length` reason and produce no final text. This failure mode is easily mistaken for a simple processing error.
2. **Unpredictable Caching Costs:** The promise of prefix caching is that repeated context is processed only once, drastically reducing cost. In practice, its effectiveness was inconsistent. While a cache hit cost $0.0466, a miss cost $0.3916—a **nearly 8x difference**. For a workload like this, where the daily token quota is a hard limit, these unpredictable cache misses could make or break a project’s budget.
3. **The Latency Trap:** Long context is not fast. One question (C2), which benefited from a perfect cache hit, still took 208 seconds to answer because the model had to “think” through over 127,000 tokens. For interactive applications, response times in the hundreds of seconds are entirely impractical.
### The Results: A Clear Winner for This Use Case
After grading, the results were decisive:
* **Long Context Performance:** Answered all 12 questions perfectly, with full scores across correctness, completeness, and groundedness. With such a small corpus relative to the context window, it didn’t need to perform any retrieval and simply had all the information available.
* **RAG Performance:** Scored nearly as well on correctness and groundedness, meaning it didn’t hallucinate facts. However, its **completeness score was very low (0.83/2)**. It frequently failed to provide the comprehensive, specific answers required by the corpus-wide questions, often giving up or being vague.
**Cost and Speed:** The long-context path was **16 times more expensive** and **3 times slower** than the RAG path for this modest workload.
### When to Use Which Approach?
So, can you skip RAG now? The answer is nuanced:
* **For Small, Private Knowledge Bases:** If your knowledge base is small (e.g., a few dozen documents), queried infrequently, and you value development speed and simplicity, **skipping RAG is a viable and often superior option.** The long-context method is far easier to implement, requiring no chunking, embedding models, or vector database maintenance.
* **For Larger-Scale or High-Volume Applications:** If you anticipate a high volume of queries (e.g., thousands per day), the cost of long-context requests will quickly become prohibitive, even with caching. In these scenarios, a well-tuned RAG system remains the more scalable and cost-effective solution.
### Conclusion
This experiment’s most valuable lesson is not just *what* it found, but *how* it found it. The initial run with “empty” answers was a humbling reminder that with modern LLMs, a lack of visible output doesn’t always mean a bug in the code—it can signal a bug in the configuration, specifically the `max_completion_tokens` limit.
Ultimately, for a corpus of this size and query volume, the author concludes that **RAG can indeed be skipped.** The long-context approach delivered complete, high-quality answers with significantly less engineering overhead. However, this conclusion is tightly bound to the specific scale and usage pattern. The critical takeaway is to understand your workload’s query frequency and to always monitor model metrics like `finish_reason` and thinking tokens. Without this visibility, you risk misinterpreting a model’s behavior and planning costs based on optimistic assumptions.
