Ndcg3 Retrieval Eval

ndcg3-retrieval-eval

Annotation-Free Reinforcement Learning Query Rewriting via Verifiable Search Reward — Cha et al. (2025) (arXiv:2507.23242, 2025)

What this evaluates

Evaluates the effectiveness of query rewriting models in retrieving relevant documents from a corpus using vector, lexical, and multimodal retrieval systems. It measures how well rewritten queries match the intended source documents across text-only and unstructured visual document benchmarks.

Datasets

• MS MARCO v2.1 testset 1% — total 1011; splits: test (1011)
• MTEB VIDORE V2 benchmark — total 327; splits: test (327)
• In-house industrial data — total 4398; splits: test (4398)

Metrics

• NDCG@3 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 3. It measures the quality of the ranked retrieval results by discounting the relevance score logarithmically with position, normalized by the ideal DCG. The paper notes it represents recall well with ordinal scoring.

Input / output format

Input: User query and a corpus of documents (vector embeddings, BM25 index, or raw unstructured documents depending on the RAG framework).

Output: Rewritten query optimized to retrieve the source corpus from the retriever.

Scoring recipe

import math

def compute_ndcg_at_3(retrieved_ids, relevant_ids):
    # Binary relevance for top-3 results
    rel_scores = [1.0 if rid in relevant_ids else 0.0 for rid in retrieved_ids[:3]]
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rel_scores))
    # Ideal DCG: all relevant docs placed at top
    ideal_rel = sorted([1.0 if rid in relevant_ids else 0.0 for rid in retrieved_ids], reverse=True)[:3]
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal_rel))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The RL training reward is binary (1 if source corpus is retrieved), but the reported evaluation metric is NDCG@3, which uses ordinal scoring over the top-3 ranked results. Do not confuse the training signal with the evaluation metric.
• Dataset sizes are reported in terms of queries, not document-query pairs. NDCG@3 should be computed per query and then averaged, not aggregated over the entire corpus.
• Three RAG backends (Semantic, Lexical, Multimodal) are described, but the excerpt does not specify whether NDCG@3 is averaged across them or reported separately. Replicators should verify if framework-specific breakdowns are required.

Evidence (verbatim from paper)

The virtue of retrieval task is to maximize recall, which NDCG represents it well with ordinal scoring. Therefore, we deploy NDCG@3 for the target evaluation metric and the reward score.

Citation

@misc{cha2025annotationfree,
  title={Annotation-Free Reinforcement Learning Query Rewriting via Verifiable Search Reward},
  author={Cha et al. (2025)},
  year={2025},
  note={arXiv:2507.23242}
}

• arXiv: 2507.23242