Evaluate an LLM judge like a versioned measurement instrument. Define one criterion at a time, anchor the rubric with examples, compare verdicts against labeled cases, test position and verbosity bias, measure disagreement, allow abstention, and replay calibration whenever the model or prompt changes. Until that evidence exists, the judge should advise—not block—a release.

Decide

Whether an LLM judge is reliable for one criterion

Artifact

A disagreement fixture for calibration

Prerequisite

Human labels with rationales and representative slices

Judge calibration loop
  1. 01
    DefineOne criterion
  2. 02
    LabelAnchors + rationale
  3. 03
    MeasureBias + disagreement
  4. 04
    OperateAbstain + monitor
Calibration is not a one-time benchmark. Disagreement and production failures return to the labeled set before the next judge version is promoted.

Criterion first

Judge one defined quality at a time#

Evidence trailSource 1Source 4

Groundedness, policy compliance and task completion are different constructs. A holistic “quality” prompt lets the model change the criterion from case to case and makes disagreement impossible to diagnose.

Give the judge observable anchors for pass, partial, fail and abstain. Ask it to identify evidence before assigning a verdict, and store the rubric version with the result.

Calibration set

Compare with labels that have a provenance#

Evidence trailSource 1Source 2Source 3

Build a compact set of clear positives, clear negatives, difficult boundaries and legitimate disagreements. Labels need a written rationale and, for high-stakes criteria, review by a domain owner.

Measure per-class behavior and disagreement rather than celebrating one aggregate accuracy. A judge can look strong overall while failing the rare slice that matters most.

Reference implementation

Inspect disagreements, not only agreement

Four cases are not a calibration study; they make the error surface visible and give the workflow a concrete starting format.

calibration.jsonl
{"case_id":"a","human":"PASS","judge":"PASS","slice":"direct"}
{"case_id":"b","human":"HOLD","judge":"PASS","slice":"missing_evidence"}
{"case_id":"c","human":"HOLD","judge":"HOLD","slice":"adversarial"}
{"case_id":"d","human":"PASS","judge":"HOLD","slice":"verbose"}

Failure exercised. Cases b and d show opposite errors. A single agreement rate hides which mistake the judge makes.

Production boundary. Use enough independent labels to estimate uncertainty; this tiny fixture is didactic, not statistical evidence.

Adversarial validation

Test the biases the literature already names#

Evidence trailSource 2Source 4
  1. 01

    Position

    Swap candidate order; a stable verdict should not follow the slot.

  2. 02

    Verbosity

    Hold meaning constant while changing length; longer must not win by default.

  3. 03

    Self-preference

    Blind model identity and compare across model families.

  4. 04

    Criteria drift

    Replay anchors throughout long runs and across rubric versions.

Uncertainty contract

Make disagreement and abstention visible#

Evidence trailSource 3

A forced answer converts missing evidence into false certainty. Give the judge an abstain path and keep its evidence separate from the final policy decision.

For important gates, use independent judgments or a small jury and inspect where they disagree. A majority vote is useful only when jurors are not copies of the same failure mode.

Change control

A new model is a new measuring instrument#

Evidence trailSource 4

Model upgrades, prompt edits, rubric changes and decoding parameters can all move the verdict distribution. Pin them, record them and replay the stable calibration set before promotion.

Monitor disagreement and slice-level errors after release. Production feedback should extend the calibration set, not silently rewrite old labels.

Validity boundary

What a validated judge can—and cannot—do#

Evidence trailSource 1Source 2Source 3Source 4

CAN

  • Scale a defined rubric over many cases
  • Expose evidence and recurring disagreements
  • Protect known behaviors in regression tests
  • Prioritize cases for human review

CANNOT

  • Become ground truth by sounding confident
  • Generalize beyond its calibration set automatically
  • Replace domain owners in high-stakes decisions
  • Eliminate bias with one prompt trick

Read next

Harness field note · 01

LLM quality gates: turn evaluations into release decisions

A practical framework for combining deterministic checks, LLM judges, trace evidence, and explicit PROMOTE, HOLD, or ROLLBACK decisions.

Release field note · 02

LLM application release checklist: evidence before traffic

A practical pre-release checklist for LLM applications covering contracts, evals, security, observability, ownership, and rollback.

Primary sources

Primary sources

  1. G-Eval

    Explicit evaluation steps and structured form-filling.

  2. MT-Bench and Chatbot Arena

    Position, verbosity, and self-enhancement biases.

  3. Replacing Judges with Juries (PoLL)

    Panels of model judges and aggregation.

  4. Who Validates the Validators?

    Criteria drift in LLM-based evaluation.

Validate the evaluator

Build judge contracts that expose evidence and uncertainty.