Scientific Relation Classification Eval

scientific-relation-classification-eval

What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification — Bassignana et al. (2022) (arXiv:2204.13516, 2022)

What this evaluates

Evaluates the robustness and cross-dataset/domain generalization of relation extraction models on scientific abstracts. It probes how annotation discrepancies and domain shifts affect relation classification performance.

Datasets

• SemEval-2018 — total ?; splits: train (257), dev (50), test (50)
• SciERC — total ?; splits: train (257), test_NLP (50), test_AI-ML (52), test_CV (105), test_SPEECH (35)

Metrics

• Macro F1-score (primary) — range: [0, 1]
  • Computes the unweighted mean of the F1-score for each relation class, then averages them across all classes. It treats all classes equally regardless of frequency.

Input / output format

Input: Scientific abstract sentences containing two entity spans and their surrounding context.

Output: A relation label from the unified label set (e.g., METHOD, TASK, METRIC) assigned to the entity pair.

Scoring recipe

def compute_macro_f1(preds, gold):
    classes = sorted(set(preds + gold))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• Cross-dataset evaluation uses overlapping abstracts with divergent annotations, so models trained on one dataset may not generalize directly to the other without handling annotation conflicts.
• The paper reports results averaged over three different seeds, so single-run evaluations will not match the reported numbers.
• SciERC is split into sub-domains (NLP, AI-ML, CV, SPEECH); cross-domain tests specifically train on NLP and test on the others, which differs from standard in-domain evaluation.

Evidence (verbatim from paper)

We use macro F1-score as evaluation metric. All experiments were run over three different seeds and the results reported are the mean.

Citation

@misc{bassignana2022survey,
  title={What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification},
  author={Bassignana et al. (2022)},
  year={2022},
  note={arXiv:2204.13516}
}

• arXiv: 2204.13516