Evaluating Retrieval Without Golden Answers, Judges and Canaries

Pull-quote: “You do not need ten thousand labeled answers to catch a broken retriever. You need a few hundred canaries you trust and an alarm wired to them.”

Why this matters

Classic retrieval evaluation assumes labeled relevance judgments: a query set, a corpus, and human annotations of which documents answer which queries. Enterprise corpora almost never have this, and the ground shifts constantly: documents arrive nightly, the question distribution drifts with the business, and every chunking or model change silently re-deals the deck. Waiting for a labeling project to finish is how retrieval regressions reach users first. The alternative is not to skip evaluation. It is to build instruments that work without golden answers.

Three instruments

Judge models. An LLM judge scores what production actually retrieved: is each passage relevant to the question, and is every claim in the answer supported by the retrieved evidence? Judges need discipline to be worth trusting. Score against a written rubric, not vibes. Prefer pairwise comparisons over absolute scores, because judges are more consistent at “which is better” than “how good is this.” And treat the judge as a model with its own failure modes: fluency preference, position bias, self-preference toward text in its own style. Swap presentation order, fix the rubric, and calibrate a sample of judge verdicts against human review on a schedule.

Canary queries. A curated probe set of questions where you know where the answer lives: not a full golden answer, just “the answer is in document X, section Y.” That weaker label is dramatically cheaper to maintain and still catches most regressions that matter: the chunking change that dropped a section, the filter bug hiding a revision, the embedding upgrade that shifted a neighborhood. Run canaries on every index rebuild and configuration change, and nightly in between.

Drift alarms. Distributions, not judgments. No labels required at all.

The operating loop

   Production queries ──► sample ──► LLM judge (rubric, pairwise)
        │                                │
        │  canary set (scheduled) ───────┤
        ▼                                ▼
   Metrics store ──► drift alarms ──► human audit queue
        ▲                                   │
        └── fixes: corpus, chunking, filters, index ◄──┘

The loop matters more than any single instrument. Judges without human calibration drift into flattery. Canaries without alarms are a dashboard nobody reads. Alarms without an audit queue produce alert fatigue. Wire them together, and the humans spend their scarce attention exactly where the instruments disagree or degrade.

Where this shows up in the field

Evidence synthesis is the hard end of the spectrum: those questions rarely have one correct passage, so pairwise judge verdicts and scheduled human calibration carry the weight, and consequential decisions keep a human in the loop by design. Document-heavy corpora sit at the tractable end: canary probes pinned to known reports are cheap to maintain, and the operating rule writes itself: an index rebuild that fails the canary set does not ship.

Closing

Golden answers are a luxury; accountability is not. A judged sample of live traffic, a canary set with known sources, and alarms on retrieval distributions will catch the regressions that matter, usually before users do. Start with twenty canaries this week. The perfect labeled benchmark you are waiting for is the reason your last regression reached production unobserved.