Proximity RAG Eval

proximity-rag-eval

Leveraging Approximate Caching for Faster Retrieval-Augmented Generation — Bergman et al. (2025) (arXiv:2503.05530, 2025)

What this evaluates

Evaluates an approximate caching system for Retrieval-Augmented Generation (RAG) pipelines. It measures how well the cache preserves retrieval quality and end-to-end accuracy while reducing database lookup latency under uniform and skewed query workloads.

Datasets

• MMLU (econometrics subset) — total 131; splits: test (131)
• MedRAG (PubMedQA subset) — total 200; splits: test (200)
• MedRAG-Zipf — total 10000; splits: test (10000)

Metrics

• test accuracy (primary) — range: percent
  • Percentage of multiple-choice questions in the query workload answered correctly by the LLM. Computed as (correct predictions / total questions) * 100.
• cache hit rate — range: percent
  • Percentage of queries that find a sufficiently similar match in the cache within the defined similarity tolerance. Computed as (cache hits / total queries) * 100.
• retrieval latency — range: other
  • Average time required to retrieve relevant documents, including both cache lookups and vector database queries where necessary. Measured in milliseconds or seconds per query.
• database k-recall — range: [0, 1]
  • Fraction of the top-k documents returned by the cache that also appear in the top-k results retrieved directly from the vector database for the same query. Computed as |top_k_cache ∩ top_k_db| / k.

Input / output format

Input: Query text (multiple-choice question) + pre-indexed document corpus (wiki_dpr or PubMed) with 768-dimensional embeddings. Queries are processed through a RAG pipeline using LLaMA 3.1 Instruct.

Output: Multiple-choice answer selection (for accuracy), cache hit/miss flag, retrieval time measurement, and top-k document indices from both cache and database for recall computation.

Scoring recipe

def compute_metrics(predictions, golds, cache_hits, total_queries, cache_topk, db_topk, k, latencies):
    accuracy = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds) * 100
    hit_rate = cache_hits / total_queries * 100
    avg_latency = sum(latencies) / len(latencies)
    recall = sum(len(set(cache_topk[i]) & set(db_topk[i])) / k for i in range(len(cache_topk))) / len(cache_topk)
    return accuracy, hit_rate, avg_latency, recall

Common pitfalls

• High similarity tolerance (tau) increases cache hit rate but can degrade test accuracy by injecting irrelevant documents into the LLM prompt.
• Database k-recall is a direct proxy for retrieval quality but does not account for downstream LLM variability or prompt sensitivity.
• The synthetic MedRAG-Zipf workload simulates query repetition bias but explicitly does not simulate temporal locality, which may overestimate cache effectiveness in real-world streaming scenarios.

Evidence (verbatim from paper)

Our evaluation focuses on three performance metrics: (i) The test accuracy of the entire RAG (retrieval-augmented generation) system, which is computed as the percentage of multiple-choice questions in our query workloads answered correctly by the LLM (large language model);(ii) The cache hit rate, which is defined as the percentage of queries that find a sufficiently similar match in the cache;(iii) The retrieval latency, which is the time required to retrieve the relevant documents, including both cache lookups and vector database queries where necessary. To quantify how well the cache preserves the quality of retrieved context, we also measure the database k-recall, defined as the fraction of the top-k documents returned by the cache that are also among the top-k results retrieved from the vector database for the same query.

Citation

@misc{bergman2025proximity,
  title={Leveraging Approximate Caching for Faster Retrieval-Augmented Generation},
  author={Bergman et al. (2025)},
  year={2025},
  note={arXiv:2503.05530}
}

• arXiv: 2503.05530