Insight · November 12, 2026
The model that
grades the model.
Code can check an exact answer. A person can judge nuance but is slow. An LLM judge reads for quality at scale, and carries a model's biases while it does.
01 · The short answer
What LLM as a judge means.
LLM as a judge is the practice of using one language model to score another model's output against a set of criteria. You hand the judge a rubric and either a single answer to rate or two answers to compare, and it returns a verdict: a number, a label, or a winner. It is how teams grade qualities that rules cannot measure, such as tone, helpfulness, and whether an answer is actually faithful to its source.
It exists because the other two options each fall short. Exact checks in code are cheap and reliable but blind to nuance. Human review is nuanced but slow. A model judge sits between them: it reads for meaning and runs on every output for a fraction of a person's cost.
02 · Why it exists
Three ways to grade an answer, and only one of them scales to nuance.
Anthropic's own evaluation guidance lays out three grading methods. Code based grading is the fastest and most reliable, and it scales without limit, but it lacks nuance for the judgments that need less rule based rigidity. Human grading is the most flexible and the highest quality, and the same guidance says to avoid it where you can, because it is slow and expensive.
Model based grading is the third method, and it is built for the work that used to require a person: rating the tone of a reply, or deciding whether a free form answer is correct. You write a grader prompt, ask for a measurable output such as a rating from 1 to 5 or a plain correct and incorrect, and run it at the scale of code with something closer to the judgment of a human. That is the whole appeal. It is also the whole risk, because the grader is itself a model.
The one line to keep
“A judge you have not calibrated is not a measurement. It is a second opinion in the clothes of one.”
03 · How a judge is built
Five steps, from rubric to a score you can trust.
01
Give it a rubric
The judge gets explicit criteria, written out: what a good answer has, what a bad one does, and what each score level means. A vague instruction to rate quality produces a vague score.
02
Show it the work
One output to score on its own, or two outputs to compare head to head. The second mode, picking a winner between A and B, is where the judge is most reliable and most biased.
03
Make it reason first
The judge explains its reasoning before it commits to a score, so the verdict is traceable to the criteria rather than pulled from nowhere.
04
Return a measurable verdict
A number on a fixed scale, or a label like correct or incorrect, not a paragraph. A score you can count is a score you can track across a thousand cases.
05
Check the judge
Before you trust it, compare its verdicts against a set a person already labeled. A judge is only as good as its agreement with the humans it stands in for.
04 · Score one, or compare two
A judge rates a single answer, or picks a winner between two.
There are two ways to use a judge, and they fail differently. The first scores one output on its own, against the rubric, and returns a rating. It is simple to run and easy to track over time, but a model's sense of an absolute score drifts. Ask it to rate the same answer on different days and the number moves.
The second shows the judge two answers and asks which is better. Comparison is the mode the research leans on, because a model is steadier at relative judgments than absolute ones, and because it maps directly onto the question teams actually ask: is the new version better than the old one. It is also the mode where the judge's worst habit lives. When you ask a model to pick between A and B, the position each answer sits in starts to sway the verdict, before quality gets a vote. That is the subject of the next section, and the reason a raw judge score cannot be taken at face value.
05 · One example that shows the shape
A rubric, reasoning first, then a verdict you can parse.
A good judge prompt does three things the guidance keeps repeating. It spells out the criteria instead of asking for a vague quality score. It tells the judge to reason before it rates, so the score traces back to the rubric. And it demands a measurable output, a number or a label, not a paragraph you then have to grade yourself.
One more rule carries real weight. Anthropic's evaluation docs put it plainly: it is generally best practice to evaluate with a different model from the one used to generate the output you are grading. A model asked to judge its own family tends to go easy on it, so the judge and the author should not be the same.
You are grading a customer support reply against a rubric.
Rubric:
· Faithful: every claim is supported by the ticket context.
· Resolving: the reply answers the actual question asked.
· On policy: no promises the policy does not allow.
First, reason step by step about the reply against each
criterion. Then return only:
{ "faithful": 1-5, "resolving": 1-5, "on_policy": "yes" | "no" }A scoring judge · explicit rubric, reasoning first, parseable verdict
06 · Where a judge goes wrong
The judge is a model, so it carries a model's biases.
The foundational study of the method, Zheng and colleagues' paper on judging the judge with MT-Bench and Chatbot Arena, named three biases that show up again and again. Position bias: shown two answers, a judge tends to favor the one in the first position, regardless of which is actually better. Verbosity bias: it tends to prefer the longer, more padded answer, which the authors demonstrated by inflating a response into a repetitive list and watching the score rise. Self enhancement bias: a judge leans toward outputs that look like its own.
None of this makes the method worthless. The same paper found that a strong judge agreed with human preferences more than 80 percent of the time, about the rate at which humans agree with one another. Excluding ties, the judge's agreement reached 85 percent, higher than the 81 percent among the people. The method works. It just works through a lens that bends in known directions, and knowing the directions is what lets you correct for them. This is the same discipline that makes evals the new unit test hold up in production.
07 · How to make it trustworthy
Calibrate the judge before you read from it.
The corrections follow straight from the biases. For position bias, run each comparison twice with the answers swapped and keep only the verdicts that hold both ways. For verbosity, make the rubric reward support rather than length. For self enhancement, judge with a different model than the one that wrote the answer. Ask for reasoning before the score, and hold the temperature low so the same input grades the same way twice.
The step that matters most is the one teams skip. Before you trust a judge, measure it against a set a person has already labeled, and treat the judge as reliable only where it agrees with those labels. A judge cannot be more aligned than the humans it stands in for, and its agreement is a number you can watch, not a thing you assume. Re measure it whenever the judge model, the prompt, or the task changes.
Used this way, a judge fits a larger loop. Anthropic describes the evaluator optimizer workflow, where one model generates a response while another evaluates it and feeds back in a loop, as effective when there are clear evaluation criteria and iterative refinement adds measurable value. A calibrated judge is what supplies those criteria. It also underwrites the grading inside evaluating a retrieval system and the runtime cousin of the method, the guardrail that screens output before a user sees it.
Closing
A judge is an instrument. Calibrate it.
Take one quality you grade by hand today. Write the rubric. Label fifty cases yourself. Point a judge from a different model at them, and keep it only where it matches you. That agreement number is the moment the judge becomes a measurement you can build on.

Written by · November 12, 2026
Donny Smith · Founder & Executive Creative Director, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInShare this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.