**Article: How to Properly Evaluate Your RAG System – Beyond RAGAS**
Modern Retrieval-Augmented Generation (RAG) systems often look functional on the surface: queries return answers, no errors are raised, and the outputs appear fluent. Yet hidden underneath are failures such as retrieving the wrong chunks, stitching together incomplete context, or hallucinating details from slightly outdated facts. These issues won’t resolve themselves—they must be surfaced through deliberate evaluation. This article walks through building a robust evaluation pipeline for production RAG systems, from basic checks to automation, custom human‑in‑the‑loop reviews, and ongoing monitoring.
What’s in this Article
– Building the Golden Dataset
– Doing a simple manual pass check before automating
– Automate scoring with RAGAS
– Adding a custom LLM judge for what RAGAS can’t check
– Using a human in the loop
– Watch for drift after shipping the system
– Running it as a pipeline
– Conclusion
Building the Golden Dataset
Before implementing any evaluation library, you need a “golden dataset”—a curated set of questions with known correct answers and the exact source document that contains each answer. This is the foundation; skipping it is the most common mistake teams make. Each entry should include:
– Question
– Ground‑truth answer (written by a domain expert)
– Source document reference
– Category (e.g., single fact, multi-hop, no‑answer expected, conflicting/stale, adversarial phrasing)
Even 20–30 well‑crafted entries can reveal meaningful patterns. Prioritize categories that expose real‑world failure modes:
– Multi‑hop (answer requires combining multiple chunks)
– No answer expected (system should refuse rather than guess)
– Conflicting or stale documents (old and new versions coexist)
– Adversarial phrasing (same intent expressed differently)
The “conflicting or stale documents” category is especially critical and often neglected. Production document stores accumulate outdated versions; without tests for them, your system will silently cite wrong but confidently sourced content.
Doing a Simple Manual Pass Check Before Automating
Automating evaluation is powerful, but it’s vital to first run the golden set manually through your pipeline. Read each generated answer alongside its ground truth before introducing scoring tools. This catches glaring issues—broken prompts, empty retrieval, ignored context—and validates that your dataset is meaningful. Automating a broken pipeline only gives you precise numbers for incorrect behavior.
Automate Scoring with RAGAS
Once basics hold, RAGAS provides fast, repeatable metrics without reading every answer. Evaluate four key dimensions:
– Context Precision: What fraction of retrieved chunks were relevant?
– Context Recall: Did retrieval capture the information needed for the answer?
– Faithfulness: Is the answer grounded in the retrieved context (i.e., no hallucination)?
– Answer Relevancy: Does the answer address the actual question?
Example code:
“`python
from ragas import evaluate
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy,
)
from datasets import Dataset
eval_dataset = Dataset.from_list([
{
“question”: item[“question”],
“answer”: rag_pipeline.query(item[“question”]),
“contexts”: rag_pipeline.retrieve(item[“question”]),
“ground_truth”: item[“ground_truth”],
}
for item in golden_set
])
result = evaluate(
eval_dataset,
metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(result)
“`
Typical output might show high faithfulness but limited context recall or precision. Importantly, faithfulness only checks whether the answer is supported by retrieved context, not whether that context is correct. A system can be fully grounded in a stale or wrong document and still score perfectly on faithfulness—this is RAGAS’s key limitation.
Adding a Custom LLM Judge for What RAGAS Can’t Check
Standard RAGAS metrics rely on generic LLM prompts that lack domain specifics. When correctness depends on exact figures, recency, required disclaimers, or tone, custom evaluation is necessary. You can use an LLM as a judge with a tailored prompt that scores these aspects directly.
Example judge prompt:
“`
Compare the generated answer to the ground truth. Score 1–5 on:
1. numeric_accuracy
2. recency_awareness
Return JSON:
{“numeric_accuracy”: int, “recency_awareness”: int, “reasoning”: str}
“`
Implementation example:
“`python
import json
from google import genai
from google.genai.types import GenerateContentConfig
client = genai.Client(
vertexai=True,
project=”YOUR_GCP_PROJECT_ID”,
location=”us-central1″,
)
JUDGE_PROMPT = “””
Compare the generated answer to the ground truth. Score 1-5 on each dimension.
Question: {question}
Ground Truth: {ground_truth}
Generated Answer: {answer}
Source Document Date: {doc_date}
1. numeric_accuracy – are all numbers and facts correct, not just plausible?
2. recency_awareness – if the source is outdated, does the answer flag
uncertainty instead of stating it as current fact?
Return JSON only:
{{
“numeric_accuracy”: int,
“recency_awareness”: int,
“reasoning”: str
}}
“””
def custom_judge(question, ground_truth, answer, doc_date, client):
prompt = JUDGE_PROMPT.format(
question=question,
ground_truth=ground_truth,
answer=answer,
doc_date=doc_date,
)
response = client.models.generate_content(
model=”gemini-2.5-flash”,
contents=prompt,
config=GenerateContentConfig(
temperature=0,
max_output_tokens=250,
response_mime_type=”application/json”,
),
)
return json.loads(response.text)
“`
Because this is slower and more costly, scope custom judging to high‑risk categories (e.g., conflicting_docs, no_answer_expected) rather than running it on every question.
Using a Human in the Loop
LLM‑to‑LLM evaluation will never perfectly align with human judgment. It’s common for judge‑to‑human agreement to sit in the low‑to‑mid 80% range—not a flaw, but a realistic ceiling to acknowledge. A practical approach is to flag cases where RAGAS and the custom judge disagree beyond a threshold for human review.
Example:
“`python
def needs_human_review(ragas_score, judge_score, threshold=1.0):
return abs(ragas_score – judge_score) > threshold
“`
This keeps the human queue small and targeted. Occasionally have two domain experts score the same borderline answers independently. If they frequently disagree, the issue is often ambiguous ground truth, not the scoring pipeline—update the golden entry, not the judge.
Watch for Drift After Shipping the System
A frozen golden set can’t catch changes in the live corpus or user query patterns. To monitor drift, sample production queries periodically (e.g., weekly) and run metrics that don’t require gold answers—context precision and faithfulness. A sudden drop in these metrics without a code change likely signals corpus drift, a failure mode missed by static evaluations.
Example sampling function:
“`python
import random
from datetime import datetime, timedelta
def sample_production_queries(logs, n=50, days=7):
recent = [q for q in logs if q[“timestamp”] > datetime.now() – timedelta(days=days)]
return random.sample(recent, min(n, len(recent)))
“`
Running it as a Pipeline
Evaluation becomes a real pipeline with two operational principles: cost-aware scheduling and CI gating:
– RAGAS on all golden set entries per PR touching retrieval or prompts
– Custom judge on flagged categories only
– Human review weekly on disagreement queues
– Production sampling weekly on live traffic
Example regression check:
“`python
def check_regression(current_scores, baseline_scores, threshold=0.03):
regressions = [
(metric, baseline_scores[metric], score)
for metric, score in current_scores.items()
if baseline_scores[metric] – score > threshold
]
if regressions:
raise SystemExit(f”Blocking merge — regressions found: {regressions}”)
print(“No regressions. Safe to merge.”)
“`
Wire this into CI so merges are blocked when metrics degrade beyond acceptable thresholds. This is the most critical safeguard against regressions reaching production.
Conclusion
No single evaluation step is impressive on its own, but together—golden datasets, manual checks, RAGAS, custom LLM judges, human review, drift monitoring, and CI gating—they form a trustworthy evaluation infrastructure. Move incrementally: start with a small golden set and manual RAGAS runs, then expand as needed. Done well, this pipeline sits quietly in CI, catching issues before they become support tickets, ensuring your RAG system remains accurate and reliable over time.
