Citebench Eval

citebench-eval

Benchmark for Evaluation and Analysis of Citation Recommendation Models — Maharjan (2024) (arXiv:2412.07713, 2024)

What this evaluates

Evaluates the capability of citation recommendation models to identify relevant academic references given local citation contexts. It probes robustness across varying contextual features, including context length, reference position, academic field, publication year, citation count, and part-of-speech tags.

Datasets

• S2ORC/S2AG Diagnostic Datasets — total ?; splits: test (-1); repo https://github.com/puzzz21/citeBench

Metrics

• Recall — range: [0, 1]
  • Fraction of ground-truth references retrieved within the top 10 predicted candidates.
• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: average of 1/rank for the first correctly ranked reference within the top 10 predictions.

Input / output format

Input: Local citation context (text snippet surrounding the citation location), optionally augmented with features like bidirectional/unidirectional context, character/word-level representations, and metadata (field, year, citation count, POS tags).

Output: Ranked list of candidate references (top-K, where K=10).

Scoring recipe

def compute_metrics(predictions, gold_refs, k=10):
    top_k = predictions[:k]
    recall = len(set(top_k) & set(gold_refs)) / len(gold_refs) if gold_refs else 0
    rr = 0.0
    for i, pred in enumerate(top_k):
        if pred in gold_refs:
            rr = 1.0 / (i + 1)
            break
    return recall, rr

Common pitfalls

• Models are trained on different datasets (e.g., Arxiv from different years), making direct performance comparisons confounded by training data distribution.
• BM25 baseline queries the entire S2AG dataset (millions of records), giving it an unfair advantage over models restricted to smaller training corpora.
• Performance heavily depends on contextual granularity (character vs. word-level) and reference position, which vary across diagnostic splits.

Evidence (verbatim from paper)

Table [1] presents the results of the models across all diagnostic datasets, highlighting the top K results where K is 10 for both Recall and MRR. BM25 shows the best performance in terms of Recall and MRR for most datasets.

Citation

@misc{maharjan2024citebench,
  title={Benchmark for Evaluation and Analysis of Citation Recommendation Models},
  author={Maharjan (2024)},
  year={2024},
  note={arXiv:2412.07713}
}

• arXiv: 2412.07713