Oag Bench Eval

oag-bench-eval

OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining — Zhang et al. (2024) (arXiv:2402.15810, 2024)

What this evaluates

This benchmark evaluates models on academic graph mining tasks, including author disambiguation, scholar profiling, entity tagging, academic recommendation, question answering, paper source tracing, and influence prediction. It probes the ability of graph neural networks, retrieval systems, and LLMs to handle structured academic data, extract attributes from long texts, and perform ranking or prediction tasks on citation networks.

Datasets

• OAG-Bench — total ?; splits: test (-1); repo https://github.com/zfjsail/OAG-Bench

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision. Computes average precision by comparing predicted importance scores [0,1] against binary labels (1 if ref-source, 0 otherwise), then averages across papers or venues.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall. Computed by comparing predicted and annotated text segments for each attribute, then averaged across attributes.
• Accuracy — range: [0, 1]
  • Overlap ratio: (1/N) * sum(|T_i ∩ T_i*| / |T_i*|), where T_i* is ground-truth tags and T_i is predicted tags.
• Recall@20 — range: [0, 1]
  • Standard recommendation metric measuring the fraction of relevant items that appear in the top-20 ranked list.
• NDCG@20 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20, measuring ranking quality with position-based discounting.
• Hit@K — range: [0, 1]
  • Binary metric indicating if the correct answer is within the top-K retrieved documents.

Input / output format

Input: Task-specific inputs: author paper lists, long-text profiles, paper attributes/venues, citation networks, research questions, and reference contexts.

Output: Task-specific outputs: anomaly scores, extracted text spans, interest tag sets, ranked recommendation lists, retrieved documents, reference importance scores, or binary award predictions.

Scoring recipe

def compute_map(predictions, golds):
    # predictions: list of importance scores [0,1] per reference
    # golds: list of binary labels (1 if ref-source, 0 otherwise)
    ap = 0.0
    hits = 0
    for rank, (pred, gold) in enumerate(zip(predictions, golds), 1):
        if gold == 1:
            hits += 1
            ap += hits / rank
    return ap / len(golds)

Common pitfalls

• Ignoring class imbalance in author disambiguation, which necessitates AUC/MAP over standard accuracy.
• Averaging metrics across attributes, scholars, or venues rather than computing per-instance scores first.
• Using standard accuracy for recommendation tasks instead of ranking-aware metrics like Recall@K and NDCG@K.

Evidence (verbatim from paper)

Evaluation Metrics. A paper may have one or more ref-sources. For each reference of the paper $p$, an importance score between $[0,1]$ needs to be output. For each paper $p$ to be traced, its reference list is encoded as 0-1 based on the labeling results (1 if it’s ref-source, 0 otherwise). By comparing the prediction result of each reference with its labeling result, we compute the Mean Average Precision (MAP). The average MAP across different papers serves as the evaluation metric.

Citation

@misc{zhang2024oagbench,
  title={OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2402.15810}
}

• arXiv: 2402.15810