Micro F1

micro-f1

ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graph — Kabongo et al. (2023) (arXiv:2305.11068, 2023)

What this evaluates

Evaluates the ability of transformer models to extract Task-Dataset-Metric (TDM) triples from scholarly AI publications. It measures how accurately models can identify leaderboard components and distinguish them from papers that do not report empirical research.

Datasets

• ORKG-Leaderboards dataset (subset from prior work [15]) — total ?; splits: test (-1); repo https://github.com/Kabongosalomon/task-dataset-metric-nli-extraction

Metrics

• micro-F1 (primary) — range: percent
  • Micro-averaged F1 score computed across all instances, treating each entity mention equally regardless of class frequency. Calculated as 2 * (precision * recall) / (precision + recall) using global true positives, false positives, and false negatives.
• macro-F1 — range: percent
  • Macro-averaged F1 score that computes the F1 score for each class independently and then takes the unweighted mean, capturing averaged class-level task evaluations.

Input / output format

Input: Plain text extracted from scholarly papers, processed via either Grobid (noisy plaintext, avg 512 tokens) or LaTeX source (clean plaintext, avg 685 tokens). Context is truncated to 512 tokens for BERT/SciBERT or 2000 tokens for XLNet/BigBird.

Output: Classification label per instance: 'TDM' (Task-Dataset-Metric triple present), 'Task', 'Dataset', 'Metric', or 'Unknown' (no leaderboard reported).

Scoring recipe

def compute_micro_f1(predictions, gold):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold):
        if p == g and p != 'Unknown': tp += 1
        elif p != g and g != 'Unknown': fn += 1
        elif p != g and p != 'Unknown': fp += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Ignoring the 'Unknown' class, which represents papers without leaderboards and is critical for emulating real-world application settings.
• Confusing macro and micro averages; macro scores average class-level performance, while micro scores reflect fine-grained instance-level discrimination and are explicitly preferred for this task.
• Context length limitations (512 vs 2000 tokens) can artificially truncate input, especially for LaTeX sources, impacting model generalization and cross-workflow comparisons.

Evidence (verbatim from paper)

Similar to our prior work [15], all experiments are performed via two-fold cross-validation. Within the two-fold experimental settings, we report macro- and micro-averaged precision, recall, and F1 scores for our Leaderboard extraction task on the test dataset. The macro scores capture the averaged class-level task evaluations, whereas the micro scores represent fine-grained instance-level task evaluations.

Citation

@misc{kabongo2023orkgleaderboards,
  title={ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graph},
  author={Kabongo et al. (2023)},
  year={2023},
  note={arXiv:2305.11068}
}

• arXiv: 2305.11068