qhjqhj00 master-set-eval
MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature — Ratul et al. (2026) (arXiv:2604.17680, 2026)
What this evaluates
Evaluates a model's ability to recommend functionally indispensable (must-cite) papers for a given query paper based only on its title and abstract. It probes scientific retrieval capability by measuring how well systems rank baseline, core-relevant, or frequently mentioned papers from a large candidate pool.
Datasets
- MasterSet-CoreML-v1 — total 74789; splits: train (67761), test (7028)
Metrics
Recall@K(primary) — range: [0, 1]- Fraction of relevant (must-cite) papers retrieved within the top K results. Calculated as |R ∩ R̂_K| / |R|, where R is the set of ground-truth must-cite papers and R̂_K is the top-K predicted list.
MAP— range: [0, 1]- Mean Average Precision across queries. Averages the precision at each relevant position in the ranked list.
MRR— range: [0, 1]- Mean Reciprocal Rank. Averages the inverse of the rank of the first relevant paper in the list.
nDCG@K— range: [0, 1]- Normalized Discounted Cumulative Gain at K. Measures ranking quality by assigning higher scores to relevant papers appearing earlier, normalized by the ideal ranking.
HR@K— range: [0, 1]- Hit Rate at K. Binary metric indicating whether at least one relevant paper appears in the top K results.
Input / output format
Input: Title and abstract of the query paper.
Output: A ranked list of candidate papers retrieved from the MasterSet-CoreML-v1 pool.
Scoring recipe
def compute_metrics(pred_list, gold_set, K):
top_k = set(pred_list[:K])
gold = set(gold_set)
recall_k = len(top_k & gold) / len(gold) if gold else 0.0
hr_k = 1.0 if top_k & gold else 0.0
# MAP, MRR, nDCG@K follow standard retrieval formulas
return {'Recall@K': recall_k, 'HR@K': hr_k}
Common pitfalls
- Using full manuscript text instead of the restricted title/abstract input violates the protocol.
- Including candidate papers published after the query paper violates the temporal split constraint.
- Treating all cited papers as positives instead of applying the three-tier must-cite criteria (baseline, core-relevance ≥4, or mention frequency ≥N).
Evidence (verbatim from paper)
Evaluation uses Recall@$K$ as the primary metric, reflecting the asymmetric cost of missing a must-cite paper. We also report MAP, MRR, nDCG@$K$, and HR@$K$ to characterise ranking quality at different depths.
Citation
@misc{ratul2026masterset,
title={MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature},
author={Ratul et al. (2026)},
year={2026},
note={arXiv:2604.17680}
}
- arXiv: 2604.17680