Leaderboard Triple Extraction Eval

leaderboard-triple-extraction-eval

Automated Mining of Leaderboards for Empirical AI Research — Kabongo et al. (2021) (arXiv:2109.13089, 2021)

What this evaluates

Evaluates a model's ability to verify whether a candidate (Task, Dataset, Metric) triple is actually used or mentioned in a specific AI research paper. The task is framed as a natural language inference problem where the model must distinguish between valid triples and randomly sampled invalid ones.

Datasets

• AI Research Paper Collection — total 4500; splits: train (-1), val (-1), test (-1)

Metrics

• micro-F1 (primary) — range: [0, 1]
  • Computes global true positives, false positives, and false negatives across all classes before calculating precision and recall. F1 is the harmonic mean of precision and recall.
• macro-F1 — range: [0, 1]
  • Calculates precision and recall for each class independently, then takes their unweighted mean. F1 is the harmonic mean of the averaged precision and recall.

Input / output format

Input: A candidate triple (t, d, m) concatenated with the paper's context feature representation (p_DocTAET).

Output: Binary classification label: true or false.

Scoring recipe

def compute_f1(y_true, y_pred, average='micro'):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Negative samples are generated by randomly selecting triples from other papers rather than negating the text, which may artificially inflate performance compared to real-world verification tasks.
• The task is multi-label and multi-class; macro-F1 will heavily penalize poor performance on rare triples, while micro-F1 is dominated by frequent ones.
• The exact construction of the DocTAET context feature (p_DocTAET) is not fully detailed in the task definition, requiring careful implementation of the underlying feature extraction pipeline.

Evidence (verbatim from paper)

Through a large-scale empirical evaluation on over 4,500 AI research papers, the approach achieves 93.0% micro-F1 and 92.8% macro-F1... The inference data instance, then is (c;[(t,d,m),p_DocTAET]) where c ∈ {true,false} is the inference label. Thus, specifically, our LEADERBOARD extraction problem is formulated as a natural language inference task between the DocTAET context feature p_DocTAET and the (t,d,m) triple annotation. (t,d,m) is true if it is among the paper's TDM-triples, otherwise false.

Citation

@misc{kabongo2021automated,
  title={Automated Mining of Leaderboards for Empirical AI Research},
  author={Kabongo et al. (2021)},
  year={2021},
  note={arXiv:2109.13089}
}

• arXiv: 2109.13089