AnthonyAlcaraz RRF Hybrid Retrieval
Overview
A DevOps incident query like "latency spike in checkout related to recent deploy" needs four kinds of retrieval:
- Semantic: conceptually similar past incidents (vector similarity)
- Keyword: exact identifiers (
checkout-api,deploy-v3.5.0) - Graph traversal: connected entities (services that depend on checkout-api; deploys that touched it)
- Temporal: recent events (last 24h, by deploy timestamp)
Any single channel misses; semantic loses exact identifiers, keyword misses synonyms, graph misses isolated nodes, temporal misses the "happened before 24h ago but matters" cases. The fusion is the point.
Reciprocal Rank Fusion (Cormack et al. 2009):
RRF(item) = sum over channels c of: 1 / (k + rank_c(item))
Where k=60 is the standard. Rank-based — no score calibration needed.
Items missing from a channel contribute zero from that channel. Items
ranked high across multiple channels naturally rise to the top.
After fusion, run a cross-encoder reranker over the top-N for precision, then apply a token-budget filter so the result fits the downstream LLM's context window. Per HINDSIGHT (Latimer et al. 2025, cited in Ch4): "After RRF, a neural cross-encoder reranker refines precision on top candidates, then token budget filtering ensures results fit the downstream LLM's context window."
When to Use
- Memory queries that span multiple retrieval-channel "shapes" (conceptual
- exact + connected + recent)
- Production DevOps incident-investigation, customer-support resolution, research-synthesis agents
- When you need to combine heterogeneous retrieval methods and don't want to calibrate scores across them
Phrases: "hybrid retrieval", "RRF", "reciprocal rank fusion", "multi- channel search", "combine semantic and keyword", "cross-encoder rerank".
When NOT to Use
- Single-channel retrieval is sufficient (e.g. keyword-only ID lookup)
- One channel dominates and the others contribute noise
- Real-time hot path where the cross-encoder rerank exceeds latency budget (~50-200ms typical) — fall back to RRF without rerank
Process
| Step | Input | Action | Output | Verification |
|---|---|---|---|---|
| 1 | query + 4 channel rankings (each a list of doc_ids in rank order) | lib.rrf_fuse(channels, k=60) |
List of (doc_id, fusion_score) sorted descending | doc_ids in multiple channels rank higher than doc_ids in one |
| 2 | top-N from fusion + query | lib.cross_encoder_rerank(top_n, query) |
Reranked top-N | reranker can re-order but does not introduce or drop items |
| 3 | reranked list + token-count fn + budget | lib.token_budget_filter(items, get_tokens, budget) |
Filtered list fitting under budget | sum(tokens) <= budget |
| 4 | full pipeline | lib.hybrid_retrieve(query, channel_callables, ...) |
Final result list | each item has fusion_score + rerank_score + token_count |
Rationalizations
| Agent rationalization | Documented rebuttal |
|---|---|
| "I'll just use vector search and call it a day." | Vector search misses exact identifiers ("incident INC-1043" — the embedding for "INC-1043" is noise). Exact identifiers are graph-traversal + keyword's job. The 4-channel composition exists because no single channel is sufficient. |
| "I'll combine scores from each channel — RRF is overkill." | Score scales differ across channels (cosine 0-1, BM25 0-100+, graph hop-count 1-10). Combining requires calibration. RRF sidesteps this by using ranks. The Ch4 anchor names this exact advantage. |
| "Cross-encoder rerank is slow — skip it." | Skip it when latency budget is tight, document the choice. The rerank produces measurable precision lift on the top-5; without it, top-5 is fusion-only and can include "popular but wrong" items. |
| "Token budget filter loses information." | Yes — that's the point. The downstream LLM's context is bounded. Either you filter, or the LLM truncates blindly. Filtering with intent (drop the lowest-rerank-score) beats truncation by accident. |
Red Flags
- RRF score distribution is flat (top-10 all very close). Channels are redundant — they're returning the same items. The fusion is buying nothing; either trim channels or check that each channel is exercising a different shape.
- Cross-encoder rerank changes top-10 by > 80%. Either the fusion is bad or the cross-encoder is mis-tuned for this domain.
- Token budget filter drops > 50% of reranked items. Budget is too small for the query complexity — either widen the budget or write multiple narrower queries.
Non-Negotiable Verification
- Run the benchmark battery.
python cli.py benchmarkmust report:- RRF formula matches Cormack et al. 2009 reference
- items in multiple channels rank above items in one
- token-budget filter respects the budget bound
- full pipeline produces results with all four score fields
- Verify CLI help. Exits 0 and prints SKILL.md description.
Security Posture
- Prompt injection. Retrieved items are untrusted content headed straight for an LLM context - fusion and rerank order them, they do not sanitize them. An adversarial document that ranks in multiple channels gets RRF-amplified into the budget-filtered prompt; run an injection filter between retrieval and prompt assembly.
- Data exfiltration. Fusion aggregates across four channels with no ACL of its own - a broad query can pull sensitive items from any channel into one context. Enforce access control per channel BEFORE fusion; the skill itself makes no network calls and writes no files.
- Privilege escalation. No shell invocation, no eval. The influence vector is rank manipulation: content stuffed with exact identifiers plus synonyms to score across channels and dominate the token budget. Cross-encoder rerank helps but is not a security control.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien) Ch4 — Retrieval section. RRF formula: Cormack et al. 2009 (Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods). HINDSIGHT 4-channel application: Latimer et al. 2025.