LLM application release checklist: evidence before traffic

01 · Behavior contract

Define success before running the model#

Write the task, allowed tools, output schema, refusal behavior, latency budget, and critical failure modes before selecting a metric. A vague requirement such as “be helpful” cannot produce an auditable gate.

Separate invariants from preferences. Schema validity, authorization and prohibited tool use are deterministic blockers. Tone, relevance and completeness may require rubrics or human review. Mixing both into one score hides severe failures.

02 · Evaluation evidence

Test known cases, boundaries, and recovery paths#

Build a versioned set with ordinary journeys, difficult boundaries, adversarial inputs and previously observed failures. Record dataset, prompt, model, tool and rubric versions with every run.

Do not promote on pass rate alone. Inspect failures by capability and risk slice, compare against the last approved release, and treat missing coverage as uncertainty rather than success.

03 · Critical blockers

Give security and authorization their own gate#

The OWASP Top 10 for LLM Applications describes risks that an average quality score cannot neutralize. Prompt injection, excessive agency, sensitive information disclosure and insecure output handling need explicit tests and owners.

A critical invariant should fail closed: the release stops even if every soft-quality indicator improves. Document the exception process so urgency cannot silently redefine the policy.

04 · Operations

Observe decisions without creating a data leak#

Trace model and tool operations, latency, errors, token use and evaluation verdicts with low-cardinality identifiers. Capture message content only when necessary, with redaction, access control and retention limits.

Connect the gate to a named release owner and a rollback signal. An alert without an owner is a log; a rollback without a tested path is a wish.

05 · Release decision

End with PROMOTE, HOLD, or ROLLBACK#

1. P

   PROMOTE

   Required evidence is complete, critical invariants pass, and residual risk is accepted by the owner.

2. H

   HOLD

   Evidence is absent, conflicting, outside the validated range, or awaiting a human decision.

3. R

   ROLLBACK

   Production evidence crosses a predefined reliability, safety, or business boundary.

{
  "candidate": "release-2026-07-20.1",
  "artifact": "image@sha256:<digest>",
  "evidence": ["tests", "evals", "security", "operations"],
  "owner": "release-owner",
  "verdict": "HOLD",
  "reason": "candidate journey not verified",
  "rollback_target": "release-2026-07-19.3"
}

Pre-traffic checklist

Twelve questions before production traffic#

• ✓Is the behavior contract versioned?
• ✓Do schema, authorization, and policy invariants pass?
• ✓Does the set cover ordinary, boundary, adversarial, and recovery journeys?
• ✓Were previous production failures replayed?
• ✓Are judge rubric and model versions pinned?
• ✓Were high-risk slices inspected separately from averages?
• ✓Does each critical risk have a hard blocker?
• ✓Are prompts and responses redacted or capture-disabled by default?
• ✓Can a run be reproduced from stored versions and identifiers?
• ✓Is there a named release owner?
• ✓Is rollback defined and rehearsed?
• ✓Is the final decision and its evidence recorded?