Primer · before you start

The vocabulary you need before Module 0.1

This course assumes you can read a Python script and that you've called an LLM before. If either feels shaky, this primer covers exactly the five ideas the rest of the course uses without explaining them again. It's free, it's short, and it exists so the modules can go straight to the method.

who this is for You can write a function in Python and you've used an LLM (ChatGPT, Claude, an API). You have not necessarily shipped an LLM to production, written an eval, or heard the words "trace", "RAG" or "gate" in this sense. If that's you, you're ready. This primer fills the gaps.

1 · Define the workflow before the judge

A judge can only inspect a boundary you have defined. Start with one recurring piece of work and write its workflow contract: who acts, what context and policy constrain the action, what observable outcome matters, and which exception returns control to a human. This prevents a fluent output from being mistaken for a useful system.

what you'll be able to do Frame one AI workflow as an inspectable contract before you write its rubric, judge, or gate.

Work: ____________________

Actor: ____________________

Context: ____________________

Allowed action: ____________________

Observable outcome: ____________________

Failure / exception: ____________________

Human owner: ____________________

Evidence retained: ____________________

a green gate is not the outcome The gate proves that an output met a versioned policy. Only the observed result after delivery tells you whether the workflow worked. Keep both: gate evidence for control, outcome evidence for learning.

which workflow-contract field stops “good writing” from becoming the success criterion?

Observable outcome. It names the result outside the model response. The judge can test a proxy before release; the real outcome is observed after the action and feeds the next calibration.

2 · An LLM call, and what a "trace" is

An LLM call is a function: you send a prompt (the input messages), the model returns a completion (the output text). That's it at the simplest level. A trace is the recorded version of that call — the input, the output, plus the things you need to debug and measure it: how long it took (latency), how many tokens it used, what tool calls happened, what context was retrieved. Every eval in this course runs on traces. No trace → no eval.

# the simplest LLM call, and what gets recorded as a trace
out = llm.chat(messages=[{"role": "user", "content": "Summarize the meeting"}])

trace = {
    "input":  "Summarize the meeting",
    "output": out.text,
    "latency_ms": out.latency,
    "tokens":    out.usage,
}
# the course's judges read fields like these — that's why traces come first

3 · RAG in two lines

RAG (Retrieval-Augmented Generation) means: before the model answers, your code fetches relevant context (chunks of documents, a knowledge base, prior messages) and puts it into the prompt. The model then answers grounded in that context instead of from memory alone. Most "hallucination" problems are really "the retrieval didn't give the model what it needed" problems. Several modules judge exactly this — whether the answer is actually supported by the retrieved context.

the one diagram to hold in your head The whole course is a loop on top of a call: you generate an output, a judge inspects it against the trace/context, a gate decides pass or fail, and on fail you retry. If you keep that shape in mind, every module is a variation on one of those four stages.

4 · The Python you'll see

You don't need to be a Python expert. You need to read it comfortably. The course uses a small, consistent subset — recognize these and you'll follow every lab:

Functions and arguments — def judge(case, llm): and keyword args like --threshold 0.7.
Dataclasses — a typed struct, e.g. @dataclass class Verdict: passed; evidence; confidence. It's just a bag of named fields.
Dicts and lists — the trace is a dict; a dataset is a list of dicts.
Running a script — python some_lab.py --input samples/. Every lab runs offline with no API key.

If statistics.median, random.seed, or a for loop look foreign, spend an afternoon on any beginner Python resource first — you'll be fine.

5 · Judge, gate, eval — the words the course uses

Three terms recur and they're not interchangeable:

A judge is code that inspects an output and returns a verdict + evidence + confidence. (Module 0.3.)
A gate is a decision that blocks or allows a release based on a judge's verdict. (Module 0.5.)
An eval is the broader practice of measuring quality systematically — traces, golden sets, drift. (Track EVAL.)

That's the entire vocabulary. The rest of the course builds on these five ideas — workflow contract, call/trace, RAG, the Python subset, and judge/gate/eval — without re-explaining them.

you don't need a labeling team One more thing, because it surprises people: this course never asks you to hand-label thousands of examples. From Module 0.4 onward you'll construct ground truth (oracles, synthetic cases, juries) so one operator sets the standard and a harness of judges enforces it. If "but who grades the answers?" is your worry — that's literally the method.

what is a "trace", and why does the course start there?

A trace is the recorded input, output, and intermediate steps of one LLM/agent run. You can't judge, gate, or eval what you didn't record — so every module downstream assumes you have traces to read.

"judge", "gate", "eval" — say each in one line

Judge: code that inspects an output and returns a verdict + evidence + confidence. Gate: a go/no-go decision that blocks or promotes based on judges. Eval: running judges over a dataset to measure quality. The course builds them in that order.

where does a judge sit in the workflow contract?

Between the candidate output and the gate. It checks the output against versioned context and policy before an action proceeds. Exceptions return control to the named human owner; observed outcomes remain a separate signal that recalibrates later cases and gates.

key takeaways

Define work, action, exception, human ownership, and observable outcome before you write a judge.
An LLM call is a function; a trace is the recorded call with the fields a judge needs. No trace → no eval.
RAG = fetch context into the prompt so the answer is grounded; most "hallucination" is a retrieval problem.
Read Python comfortably (functions, dataclasses, dicts, running a script) — that's the bar.
Judge (inspects) · gate (blocks/allows) · eval (measures systematically). The whole course is a generate→judge→gate→retry loop.