Adaptive Query Routing Eval

adaptive-query-routing-eval

Adaptive Query Routing: A Tier-Based Framework for Hybrid Retrieval Across Financial, Legal, and Medical Documents — Hashmi (2026) (arXiv:2604.14222, 2026)

What this evaluates

Evaluates retrieval and answer generation methods across structured financial, legal, and medical documents. It probes how well different architectures handle varying query complexities, cross-references, and domain-specific structural requirements.

Datasets

• Controlled Multi-Domain Corpus — total 22; splits: test (22)
• FinanceBench — total 50; splits: test (50)

Metrics

• Quality (primary) — range: [0, 1]
  • LLM-as-Judge evaluation score on a 0-1 scale assessing answer correctness and relevance.
• Recall — range: [0, 1]
  • Proportion of relevant document sections correctly retrieved by the system.
• Precision — range: [0, 1]
  • Proportion of retrieved sections that are relevant to the query.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall.
• Latency — range: seconds
  • End-to-end retrieval and generation time measured in seconds.

Input / output format

Input: Natural language queries across financial, legal, and medical domains, categorized into four complexity tiers (simple, multi-section, cross-reference, multi-document synthesis).

Output: Retrieved document sections and generated answers.

Scoring recipe

precision = len(retrieved & relevant) / len(retrieved)
recall = len(retrieved & relevant) / len(relevant)
f1 = 2 * (precision * recall) / (precision + recall)
latency = end_time - start_time
quality = llm_as_judge_score(answer, query, context) # 0-1 scale

Common pitfalls

• Assuming vector similarity reliably captures cross-references; it misses ~8.3% due to low semantic overlap with target sections.
• Treating performance as uniform across domains; Vector RAG leads in financial but lags in legal/medical where structural navigation is key.
• Ignoring latency trade-offs; reasoning-based methods incur ~2x latency for marginal quality gains on simple queries.

Evidence (verbatim from paper)

Both Tree Reasoning and Hybrid AHR achieve perfect section recall (1.00), correctly following internal references to appendices, exhibits, and notes. Vector RAG achieves 0.917 recall, missing cross-referenced sections with low semantic similarity to the query.

Citation

@misc{hashmi2026adaptive,
  title={Adaptive Query Routing: A Tier-Based Framework for Hybrid Retrieval Across Financial, Legal, and Medical Documents},
  author={Hashmi (2026)},
  year={2026},
  note={arXiv:2604.14222}
}

• arXiv: 2604.14222