How to write effective LLM as a Judge?

I wrote earlier about what to test in a RAG system and how you can use LLM as a Judge. This post is about the harder part, which is writing an LLM as a Judge you can actually act on.

Quick definition of what a judge is

An LLM scoring another model’s output against criteria you write, instead of a human rater or a fixed metric. As simple as that.

Traditional metrics fix their criteria before you run them. BLEU and ROUGE compare tokens against a reference (Ground Truth), which fails for open-ended generation where there is no single correct phrasing.

A judge is a prompt, so you change what it measures by changing what you wrote.

GPT-4 as a judge agreed with the human majority vote 85% of the time on their data. (Source : MT-Bench paper), and, Humans agreed with each other 81%.

Three types:

Pairwise: which of these two outputs is better.
Pointwise: score a single output. But absolute scores drift more when you swap judge models.
Reference-guided: score against a gold answer. Most reliable, least usable in production, because someone has to write a gold answer for every query.

Most production evals end up pointwise and reference-free, because nothing else scales. Yes the reason is that simple, but is this what you should do? 🤷‍♂️

Designing the judge

Use binary pass/fail

Sometimes people use a 5-pointer scale, but the difference between a 3 and a 4 is not a clear thing to explain. It is not clear between two annotators, and it is not clear across two runs of the same model. Both humans and LLMs drift toward the middle of a scale when they want to avoid committing to a call.

Decompose before you judge

A judge reading a whole paragraph and returning one number gives you a noisy score. Break the output into smaller claims first, judge each claim on its own, then aggregate as a ratio.

TruLens splits the response into claims and drops the non-factual ones before groundedness scoring runs. RAGAS does the same thing under the name faithfulness: extract statements, entailment-check each against the context, score the fraction supported.

Make it reason first

G-Eval works like this: have the judge generate its own evaluation steps from the criteria, then apply those steps to score. It specially helps with judging summarization kinda problems.

MT-Bench found chain-of-thought cut judge failures on math questions from 14/20 to 6/20, and to 3/20 once a reference answer was added. But focus on your use-case, not everything shown in papers or blogs (even mine!) works well in practice.

Build the rubric from actual failures

You cannot write a good judge prompt before you have looked at the data. People need to grade outputs to work out what their criteria are, so the criteria cannot be fully specified before grading starts. Let that sink in, read it twice, PLEASE.

Pick one domain expert, have them make binary calls on actual examples and write a critique explaining each call, then build the judge prompt using their actual critiques as few-shot examples. Iterate until the judge and the expert converge. This is where the Subject Matter Experts (SMEs) shine! Hehehe

Validate it before you trust it

Measure true positive rate and true negative rate against a held-out human-labeled set (SME labeled). Raw agreement reads high but tells you little when your classes are imbalanced. If more than one human is labeling, use Cohen’s Kappa.

Spend the money in the right order

Code assertions and regex first. They are deterministic, and cover a lot of known failures. A judge costs atleast 50+ labeled examples to build, non-determinism, trial and testing. Spend that only on things which require intelligence to judge, and you don’t have a set of actual answers.

Biases you are designing against

Position bias: Judges give in-consistent verdicts when the two answers are swapped. They favour whichever answer came first more than 50-60% of the time. It gets more visibility on open-ended categories like writing, but not much on math and coding.

Verbosity bias: In the MT-Bench paper, they padded answers with rephrased duplicate content that added no new information. Earlier versions of Claude, GPT, preferred the padded version 91.3% of the time.

Self-preference: Models tend to score there own outputs higher.

Judges can also be gamed directly. Bare punctuation or a generic opener like “Let’s solve this step by step” triggered false positive rewards up to 80% of the time. So basically a judge that scores well on your validation set is not automatically robust to degenerate input. So focus on edge cases while evaluating your judges as well.

The mitigations are simple enough. Swap positions and only count a win that holds both ways. Write specific criteria so that there is no or little room to interpret. Tell the judge explicitly to be strict, since scores tend to inflate by default. Give it an instruction in a way, so it can return “cannot determine” instead of guessing.

What to actually put in the prompt for RAG metrics

Context relevance. Input is the query plus one retrieved chunk, scored one chunk at a time. No response involved. Scoring the whole retrieved set at once doesn’t help you find which specific chunk was irrelevant.

Groundedness. Input is one claim from the response, checked against the entire retrieved context. Reasoning is required here, not optional.

Answer relevance. Input is the question and the response. One caveat, a confidently wrong answer can score as fully relevant if it addresses what the question asked for. Groundedness is the metric that catches the wrongness, so you need both.

None of the three needs a human-written reference answer, which is why they are getting popular in the industry.

What it comes down to

Most of the value here is not the judge. It is the process of looking at your outputs closely enough to write the rubric, and then checking whether your judge agrees with you.

Sources

MT-Bench Paper : https://arxiv.org/abs/2306.05685
G-Eval : https://arxiv.org/abs/2303.16634
RAGAS : https://arxiv.org/abs/2309.15217
Self-Preference Bias : https://arxiv.org/abs/2410.21819