Qder Re Ranking Eval

qder-re-ranking-eval

QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking — Chatterjee et al. (2025) (arXiv:2510.11589, 2025)

What this evaluates

Evaluates the ability of neural re-ranking models to effectively re-order a candidate set of documents based on complex query semantics and entity relationships. It probes fine-grained semantic matching, entity-aware attention, and late aggregation capabilities in information retrieval tasks across news and complex answer domains.

Datasets

• CODEC — total 729824; splits: test (-1)
• TREC Complex Answer Retrieval (CAR) 2017 — total 4862; splits: train (-1)
• TREC Robust 2004 — total 528024; splits: test (-1)
• TREC News 2021 — total 728626; splits: test (-1)
• TREC Core 2018 — total 595037; splits: test (-1)

Metrics

• nDCG@20 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20. Calculated as DCG@20 divided by the ideal DCG@20, where DCG sums relevance scores discounted logarithmically by position: sum((2^rel - 1) / log2(pos + 2)).
• Precision@20 — range: [0, 1]
  • Fraction of relevant documents in the top 20 ranked results.
• MAP — range: [0, 1]
  • Mean Average Precision across all queries, averaging the precision at each relevant document's rank.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, averaging the inverse rank of the first relevant document across queries.

Input / output format

Input: Query (title, description, and/or narrative fields) paired with a candidate set of 1000 documents retrieved via BM25+RM3.

Output: A ranked list or relevance score for each of the 1000 candidate documents.

Scoring recipe

def compute_ndcg_at_20(gold_rels, pred_scores, k=20):
    ranked = sorted(range(len(pred_scores)), key=lambda i: pred_scores[i], reverse=True)[:k]
    dcg = sum((2**gold_rels[i] - 1) / (2**(j+1) - 1) for j, i in enumerate(ranked))
    ideal = sorted(gold_rels, reverse=True)[:k]
    idcg = sum((2**ideal[j] - 1) / (2**(j+1) - 1) for j in range(k))
    return dcg / idcg if idcg > 0 else 0.0
# Note: Paper uses official trec_eval tool with -c flag for all metrics.

Common pitfalls

• The paper mixes re-ranking baselines (e.g., ColBERT, RoBERTa) with full-retrieval baselines (e.g., SPLADE, ANCE-MaxP) only on TREC Robust 2004, which can skew cross-dataset comparisons if not accounted for.
• Training negatives are sampled exclusively from the BM25+RM3 candidate set and balanced 1:1 with positives, differing from standard in-batch or hard-negative mining strategies.
• Significance testing relies on paired t-tests over queries, which may not fully capture variance in graded relevance judgments used by trec_eval.

Evidence (verbatim from paper)

Evaluation Metrics. (1) Precision at $k=20$, (2) Normalized Discounted Cumulative Gain (nDCG) at $k=20$, (3) Mean Average Precision (MAP), and (4) Mean Reciprocal Rank (MRR). We use the official trec_eval tool from NIST (with the -c flag) to evaluate each system.

Citation

@misc{chatterjee2025qder,
  title={QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking},
  author={Chatterjee et al. (2025)},
  year={2025},
  note={arXiv:2510.11589}
}

• arXiv: 2510.11589