AllanWessels Hybrid Retrieval
When to use
Add hybrid retrieval when /reports/latest.json shows either:
- Paraphrase-brittleness failures with high BM25 baseline scores — vector search is missing things keyword search would catch
- Exact-token failures where queries reference specific identifiers (product names, error codes, version strings) and vector recall is low
If the report shows that retrieval is working but generation is the problem, this skill is the wrong place to look.
Why hybrid beats either alone
Pure vector search and pure keyword search fail in opposite ways:
- Vector struggles when exact tokens matter — error codes, identifiers, product names all embed to roughly the same region
- Keyword struggles with paraphrase — "cancel my subscription" finds nothing if the doc says "terminate your plan"
A hybrid system runs both in parallel and merges. The merge does not need calibrated scores from either side, only ranks.
Reciprocal Rank Fusion (RRF)
For each candidate document, sum across all retrieval methods:
score(doc) = sum over methods of: 1 / (k + rank_in_method)
k=60 is the conventional constant. A doc ranked 1st in both lists
scores 1/61 + 1/61 ≈ 0.033. A doc ranked 1st in only one list scores
1/61 ≈ 0.016. Documents that appear in both lists, even at moderate
ranks, beat documents that appear in only one.
Implementation procedure
Add a BM25 index alongside the vector index. Use
rank_bm25(pure Python, ~50 lines of integration). Index the same chunks that are in the vector store, so document IDs align.Wire both into
/pipeline/query.py. At query time:- Run BM25, get top-N1 ranked list (default N1=50)
- Run vector search, get top-N2 ranked list (default N2=50)
- Apply RRF over the union, sorted by RRF score
- Take top-K of the merged list (default K=20) before reranking
Tune the lane sizes. N1 and N2 can be tuned but defaults are usually fine. The reranker downstream is what really decides what the LLM sees; the lanes' job is just to cast a wide net.
Validate per failure category, not just overall. Hybrid helps exact-token failures a lot, paraphrase failures a little. If overall score moves but a specific category gets worse, investigate.
Gotchas
- Stop words — BM25 weighting can be skewed by short queries containing only common words. Either run a stopword filter or accept that vector dominates on short queries
- Tokenization mismatch — if BM25 tokenizes "v2.0" differently from the embedding model, recall on version strings can underperform. Either share a tokenizer or pre-process queries
- Performance — running both lanes is roughly 2x the latency unless
you parallelize. In Python,
asyncioorconcurrent.futuresmakes this trivial; do it from day one
Validation hypothesis to log
When you commit a hybrid retrieval change, the changelog entry should specify:
- which failure category you expected to fix
- expected magnitude of improvement
- expected null result for unrelated categories
If your measured result diverges substantially from the hypothesis, that's a signal something else is going on.