Citation Rec Eval

citation-rec-eval

Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild — Goyal et al. (2026) (arXiv:2603.17361, 2026)

What this evaluates

Evaluates the ability of citation recommendation systems to retrieve and rank relevant academic papers given a query context. It probes ranking quality, recall of relevant candidates, and normalized discounted cumulative gain across multiple academic datasets.

Datasets

• ACL-200 — total ?; splits: test (-1)
• FullTextPeerRead — total ?; splits: test (-1)
• Refseer — total ?; splits: test (-1)
• arXiv — total ?; splits: test (-1)
• ArSyTa — total ?; splits: test (-1)

Metrics

• MRR (primary) — range: percent
  • Mean Reciprocal Rank: the average of the reciprocal of the rank of the first relevant document in the ranked list for each query.
• Recall@K — range: percent
  • Recall@K: the fraction of relevant documents found within the top K retrieved results, averaged over queries.
• NDCG@K — range: percent
  • Normalized Discounted Cumulative Gain@K: measures ranking quality by assigning higher scores to relevant documents appearing higher in the list, normalized by the ideal DCG.

Input / output format

Input: Citation context or paper text used as a query to retrieve candidate references from a corpus.

Output: A ranked list of candidate papers.

Scoring recipe

def compute_metrics(ranked_list, relevant_docs, K):
    # MRR
    rr = 1.0 / (ranked_list.index(relevant_docs[0]) + 1) if relevant_docs[0] in ranked_list else 0.0
    # Recall@K
    recall_k = len(set(ranked_list[:K]) & set(relevant_docs)) / len(relevant_docs)
    # NDCG@K
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(ranked_list[:K]) if doc in relevant_docs)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_docs), K)))
    ndcg_k = dcg / idcg if idcg > 0 else 0.0
    return rr, recall_k, ndcg_k

Common pitfalls

• The evaluation enforces a strict inductive setting with temporal constraints, preventing models from using future papers or undefined task-specific parameters.
• Metrics are reported as percentages but represent standard IR scores; Recall@K and NDCG@K must be computed at multiple cutoffs (5, 10, 20, 50, 300) as specified in the tables.

Evidence (verbatim from paper)

To provide a multi-faceted assessment of ranking performance, we employ a suite of standard information retrieval metrics (%), namely, Mean Reciprocal Rank (MRR), Recall@K, and NDCG@K.

Citation

@misc{goyal2026publicprofile,
  title={Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild},
  author={Goyal et al. (2026)},
  year={2026},
  note={arXiv:2603.17361}
}

• arXiv: 2603.17361