Module 0.1 · the method

Start from failures — triaged by a judge

Before you write a single judge, you need to know how your system fails. The state of the art says: look at your data first. The catch — you can't read ten thousand traces by hand. So a judge reads them; you read the judge's clusters and decide what matters. That's the harness in one move.

what you'll be able to do By the end of this module you can map your system's real failure modes with a discovery judge and decide which ones deserve a gate.

The trap: shipping judges for failures you never mapped

Most teams jump straight to "let's add an eval for hallucination." But hallucination might be 3% of your real failures, while silent tool-call errors are 40%. Write judges before you map failures and you harden the wrong wall. The first job is always the same: find the failure modes that actually occur, ranked by how often they occur.

the move Point a discovery judge at a sample of production traces. Its job isn't pass/fail — it's to cluster failures into named modes with a count. The output is a ranked taxonomy, not a score. You now know where to spend.

This is where the human belongs

Reading 10,000 traces by hand doesn't scale — that's a whole QA team's month. But reviewing twelve failure clusters and deciding "these three are real, that one is a labeling artifact, ignore the rest" takes one person twenty minutes. The judge does the volume; you do the judgment about what deserves a gate. This is the lonely-wolf pattern: one operator's taste, applied at machine scale.

not human-out-of-the-loop The harness doesn't remove you — it does the review work that never scaled: reading every trace, running every check, on every change. You still set direction: which clusters are real, which matter, where the bar sits. What you stop doing is the manual, repetitive reading that never scaled anyway.

our running example — the retrieval-grounded support bot Through modules 0.1→0.9 we'll carry one toy system so the method reads as a single cumulative story, not nine unrelated tricks: a retrieval-grounded support bot that answers customer questions from a knowledge base. This is the system whose traces you're about to mine. Its failure modes — un-cited answers, silent tool-call errors, over-refusals — are the ones the discovery judge below surfaces. In 0.3 you'll write judges for them; in 0.5 you'll gate on them; in 0.9 you'll wire the whole harness around them. (A pattern, not a benchmark — you build and measure your own.)

before you read on — what would you point a discovery judge at first?

A sample of production traces (input + output + any tool/error metadata), not a curated eval set. Discovery is about how the system actually fails in the wild; a hand-picked set hides the 40% mode and over-weights the 3% one.

A discovery judge, concretely

It ingests traces, proposes failure-mode labels, assigns each trace to a mode (or "clean"), and returns the taxonomy with counts and one example per mode — the evidence you'll skim.

The ranking is the output: you gate the 40% mode, not the 3% one — frequency paired with severity, never counts alone.

def discover_failures(traces, discovery_judge):
    labeled = []
    for t in sample(traces, n=500):          # judge reads volume; you won't
        mode = discovery_judge(t)             # {"mode": "silent_tool_error", "evidence": span}
        labeled.append(mode)
    taxonomy = cluster(labeled)               # ranked: mode -> (count, example)
    return taxonomy                           # YOU review this — 12 rows, not 10k traces

Now — and only now — you pick the top modes and, in the next modules, turn each into a judge with a verdict, evidence and a threshold. Discovery ranks the work; the harness does the work.

counts are the start, not the answer A ranked taxonomy tells you how often each mode happens. The next step is stratified metrics — measure precision/recall per cluster, not one global number — and an error analysis on the top clusters: are these the same root cause split into three labels, or three genuinely different failures? A frequent cluster can still be low-severity; a rare one can be the one that loses customers. Discovery gives you the map; stratification tells you which roads to pave first.

case in practice In a solo-run video pipeline the same move works: a discovery pass surfaces the failure modes that actually recur — lip-sync drift, persona breaks — and one operator decides which become hard gates. One person's standard, a panel of judges enforcing it at scale. (This is a pattern, not a benchmark — you build and measure your own in the lab.)

why discovery is a recurring ritual, not a one-off — criteria drift A discovery judge is itself an LLM, and LLM judges do not hold their criteria fixed. Shankar et al., "Who Validates the Validators?" (arXiv 2404.12272, UIST '24), document criteria drift: LLM-as-judge evaluators inconsistently apply and shift their rubric while annotating, drifting out of alignment with the human preference they were meant to encode. The consequence is direct — you cannot run discovery once and trust the taxonomy forever. Re-run it on a cadence; treat the ranked clusters as a snapshot that decays.

why cluster failures before writing any judge?

Because you harden the wall that's actually falling. Writing judges first hardens a frequency-3% mode while a 40% mode ships. Discovery ranks the work — and since a discovery judge also drifts (Shankar et al., 2404.12272), it ranks it on a cadence, not once.

a cluster has 40% of failures but is low-severity; a 3% cluster loses customers. Which do you gate first?

Neither decision follows from the count alone. Discovery gives you the map; you still pair frequency with severity (and with stratified precision/recall per cluster) to decide what earns a gate. Counts are the start, not the verdict.

▶ runnable lab

failure-discovery

Point a discovery judge at a folder of traces and get back a ranked failure taxonomy with counts and examples. Run it on the included sample, then swap in your own traces.

labs/failure-discovery/ · python discover.py

key takeaways

Map failures before you build judges — harden the wall that's actually falling.
A discovery judge clusters at scale; you review the clusters, not the raw traces.
Counts are the start — add per-cluster (stratified) metrics and an error analysis before you decide what to gate.
The human sets the bar (what's real, what matters); the judges apply it to everything.