Hybrid Retrieval, The Boring Stack That Wins
Pull-quote: “BM25 is a 1994 algorithm that keeps outperforming things invented last quarter on the queries your users actually type. That should tell you something about your users’ queries.”
Why this matters
Every few months a new retrieval technique arrives with a benchmark chart. Meanwhile, the stack that keeps winning in production is aggressively boring: BM25 for lexical matching, a dense vector index for semantic matching, reciprocal rank fusion to merge them, and a cross-encoder to rerank the merged list. It is unfashionable, well understood, and very hard to beat on real enterprise traffic. Understanding why is more useful than the recipe itself.
Why lexical still matters
Dense embeddings compress meaning into a few hundred dimensions. That compression is exactly what makes them fail on queries where the surface form is the meaning:
| Query feature | Example | Dense behavior | BM25 behavior |
|---|---|---|---|
| Identifiers | part numbers, case IDs, tail numbers | Nearest neighbors, not the ID | Exact hit |
| Rare exact terms | drug names, statute citations | Diluted by context | Strong IDF signal |
| Negation | “without corrosion” | Embeds close to “with corrosion” | Neutral, at least not misleading |
| New vocabulary | terms coined after embedding training | Out of distribution | Works immediately |
None of this means dense retrieval is weak. It is unbeatable for paraphrase and concept matching, where BM25 sees no overlapping terms at all. The two systems fail on disjoint query populations, and that disjointness is the entire argument for hybrid: you are not averaging two mediocre retrievers, you are covering two different failure sets.
Fusion and reranking
Query ──┬──► BM25 index ───────► ranked list A ──┐
│ ├──► RRF fusion ──► top ~50 ──► cross-encoder ──► top k
└──► Dense index ──────► ranked list B ──┘ (query, passage)
scored jointlyReciprocal rank fusion (RRF) merges the lists by rank position, not by score. That matters because BM25 scores and cosine similarities live on incomparable scales, and score normalization schemes are fragile across query types. RRF has one hyperparameter and almost nothing to misconfigure.
The fused list is a candidate set, not an answer. The cross-encoder reads the query and each candidate passage together and scores actual relevance, something neither BM25 nor a bi-encoder can do, since both score query and document representations built independently. Reranking the top 50 candidates is typically the single largest quality improvement available per engineering hour spent. It is also where negation finally gets handled, because the cross-encoder sees both texts in full.
Safety and regulatory corpora show why this layering matters: real query traffic mixes exact report identifiers with loose natural-language descriptions of failure modes, precisely the bimodal split that hybrid retrieval exists to cover. The lexical side also costs almost nothing to add: it indexes the same corpus, needs no training, and fails transparently enough to debug by reading the query.
Evaluate retrieval on its own
One discipline separates teams who tune this stack well from teams who thrash: a labeled retrieval set of real queries mapped to the passages that answer them, evaluated with recall@k and MRR before any LLM is involved. Generation quality is a separate measurement with separate failure modes. When the end-to-end answer is wrong, the labeled set tells you in minutes whether retrieval or generation failed. Without it, every regression becomes an argument.
Closing
Hybrid retrieval wins because enterprise queries are bimodal: half of them are secretly exact-match problems wearing natural-language clothes, and half are genuinely semantic. BM25 covers the first population, dense vectors the second, RRF merges them without score gymnastics, and a cross-encoder fixes the ordering. Boring, layered, measurable: that is what production means.