Evals Are the New Tests, Ship the Harness Before the Feature

Pull-quote: “Nobody ships a service without tests. Teams ship LLM features without evals every day, and the difference is worse than it sounds: untested code fails loudly, unevaluated models fail politely.”

Why this matters

Every input to an LLM feature is a variable: the model version, the prompt, the retrieval index, the temperature, the tool schemas. Change any one and behavior shifts, usually somewhere you were not looking. Conventional tests cannot catch this because the failure is not an exception; it is a fluent, confident answer that is slightly worse than last week’s. The only defense is an evaluation harness that runs before every change lands, which means the harness is not a follow-up task. It ships before the feature.

Separate the layers or debug forever

A single end-to-end score tells you something broke, never what. Split the harness into three layers with independent datasets and metrics:

When end-to-end drops but retrieval holds, you have a generation problem and you skip a day of index archaeology. The layers are the stack trace.

Golden sets are the asset

The golden set is a small, versioned collection of inputs with known-good outputs or labeled judgments. Start with 50–200 cases; coverage beats volume. Three sourcing rules keep it honest:

1. Seed from real traffic, not from what the team imagines users ask.
2. Promote every production failure into a case. The golden set is where incidents go to become regression tests.
3. Version it like code. A golden set that changes silently invalidates every historical score.

Calibrate the judge

Human review does not scale to every commit, so most harnesses use an LLM as judge. The judge is a model, which means it has its own error distribution, and an uncalibrated judge is a random-number generator with good grammar. The discipline: have humans label a sample of the same outputs, measure agreement (Cohen’s kappa or plain agreement rate), and recheck on a schedule and after any judge-model change. If agreement drops, fix the rubric or the judge before trusting another score. The stance is the same one that makes forecasting systems trustworthy, where predictions ship with calibrated intervals rather than bare point estimates: a score without a known error bound is not a measurement.

Wire it into CI

  Change lands (prompt, model, index, tool schema)
        │
        ▼
  ┌───────────────┐   ┌────────────────┐   ┌────────────────┐
  │ Retrieval evals│──►│ Generation evals│──►│ End-to-end evals│
  │ recall@k, MRR  │   │ judged vs golden│   │ task success    │
  └───────────────┘   └────────────────┘   └────────────────┘
        │                     │                     │
        └──────────── compare to baseline ──────────┘
                              │
                    pass ─────┴───── regression → block merge

Treat eval scores exactly like a failing unit test: a regression beyond threshold blocks the merge. Keep the CI subset small enough to run in minutes and run the full suite nightly.

Eval-driven development

The harness changes how features get built, not just how they get guarded. Write the eval cases first, watch them fail, then iterate the prompt or pipeline until they pass, and every later change is judged against the same bar. It is test-driven development with a probabilistic assert, and it converts “the model seems better” into a number someone can defend in a review.

Closing

Evals are not a QA phase; they are the definition of done. Build the golden set from real traffic, split retrieval from generation from end-to-end so failures localize, calibrate the judge against human labels before trusting it, and gate CI on regressions. Teams that ship the harness first move faster afterward, because every subsequent change comes with an answer to the only question that matters: did it get worse?