**Building a Reliable LLM Judge: Lessons from a Production Incident**
In a production system that uses autonomous agents, one common pattern is to let one Large Language Model (LLM) generate SQL queries from user questions, and then have a second “judge” agent decide whether the generated query is safe and correct enough to run automatically. This architecture sounds efficient—and it can be—but as we learned the hard way, it introduces subtle failure modes that can quietly undermine trust in the system.
### The Incident and the First Assumption
Everything started when a query that should have been flagged slipped through the judge and executed automatically. Although nothing catastrophic happened—no data loss, no writes to the wrong table—the returned result was simply wrong. The query had silently dropped a filter clause that was clearly implied by the user’s question, and it was delivered with unwarranted confidence. It took a confused half-hour with an analyst to realize the number was incorrect.
My first assumption was that the judge had made a one-off mistake. After all, LLMs aren’t perfectly reliable. But when I re-ran the same query through the same judge in isolation, the model approved it again with the same missing filter and the same confidence. This wasn’t a random slip—it was a pattern. Only then did we pull a batch of past judge decisions and compare them against what a human reviewer would have said, which revealed a structural problem.
### What the Judge Was Actually Doing
At the time, the generator agent and the judge agent were built on the same underlying model, primarily for cost reasons. When we swapped in queries generated by a *different* model—keeping task and schema identical—the judge’s behavior changed. It became noticeably stricter and caught issues that its own model’s queries had previously slipped through.
This effect is known as **self-preference bias**: an LLM tends to rate its own outputs (or outputs from its own model family) more favorably, often because they have lower perplexity and feel more familiar. What surprised us was how consistently this bias operated. The judge wasn’t random or lenient in general; it was systematically more comfortable with a specific stylistic “look and feel.”
Here’s a simplified version of the prompt that was being used at the time:
“`python
JUDGE_PROMPT = “””
You are reviewing a SQL query generated for the following user question.
Approve it for automatic execution, or flag it for human review.
User question: {question}
Generated SQL: {sql}
Schema: {schema}
Return JSON only:
{{“decision”: “approve” }}
“””
“`
Nothing in this prompt explicitly encouraged leniency, yet the bias emerged not from instructions but from the judge’s familiarity with a particular model’s output style.
### How a Judge Can Lie to You
Once we identified self-preference bias, we started looking for other systematic ways the judge could be wrong. Three recurring patterns stood out:
– **Self-preference bias**: The judge favors outputs from its own model family because they are more predictable and lower in perplexity.
– **Verbosity bias**: When asked to compare answers, the judge often prefers longer, more detailed responses—even when they are not more correct—especially if the rubric contains terms like “thoroughness.”
– **Position bias**: The order in which outputs are presented can sway the judge’s decision, particularly in pairwise comparisons.
These biases don’t mean the judge approach is invalid. Instead, they show that treating a judge’s output as an objective pass/fail test is misleading. An LLM judge is best understood as a consistent but biased reviewer whose blind spots need to be mapped and managed.
### The Fix That Helped
The single most effective change was to ensure the judge used a *different* model family than the generator. By routing judgment to a neutral third party—rather than a model that shares lineage with the generator—we directly addressed self-preference bias:
“`python
def judge_query_neutral(question, sql, schema, generator_model, client):
# Never let the judge share a model family with the generator.
judge_model = “gemini-2-5-pro” if “gpt” in generator_model else “gpt-4o”
return judge_query(question, sql, schema, client, model=judge_model)
“`
This fix solved one category of bias but left others untouched. Verbosity bias, for example, persisted because it stemmed from the rubric, not the model identity. Rewriting the rubric to explicitly penalize unnecessary length—and including a concrete example of a concise correct query outperforming a longer one—proved far more effective than generic instructions to “be objective.”
### Calibrating the Judge Against Humans
None of these changes matter without ongoing measurement against human judgment. We sampled past judge decisions and had a schema-knowledgeable human reviewer score the same queries blind. Agreement between the judge and the human was in the low-to-mid 80s percent—useful, but uneven.
We used these results to define categories where judge–human agreement was weak and automatically routed those queries to a human review, regardless of the judge’s confidence. This turned the judge into a fast first-pass triage tool while ensuring that the most error-prone cases always received a human second look. Used this way, an LLM judge can help bootstrap a labeled dataset through a human-in-the-loop process rather than replacing human judgment outright.
### Conclusion
The query that slipped through wasn’t the result of a badly written prompt. It happened because we had built a review step and quietly assumed it was neutral, when in reality it was structurally inclined to approve outputs that looked like its own. The fix that actually mattered was simple: never let the judge and the generator share the same model family.
Beyond that change, we rewrote the rubric to explicitly penalverbosity and implemented a human calibration loop that treats the judge’s output as a first opinion rather than a final verdict. These steps transformed the judge from a vague “usually fine” safeguard into a system with known, bounded failure modes—ones we’ve consciously decided we can live with.
If there’s one takeaway, it’s that an LLM judge should earn the right to gate production actions the same way a new team member does: not on the first day, and not merely because it sounds confident when it explains itself. It earns that right only after you’ve observed how it behaves, where it disagrees with humans, and exactly where its trustworthiness stops and its bias begins.
