Scienceie Keyphrase Relation Eval

scienceie-keyphrase-relation-eval

SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications — Augenstein et al. (2017) (SemEval 2017, 2017)

What this evaluates

This benchmark evaluates systems on mention-level keyphrase extraction and semantic relation extraction from scientific publications. It probes the model's ability to identify, classify, and link keyphrases across different scientific domains using exact match criteria.

Datasets

• ScienceIE — total 500; splits: train (-1), dev (-1), test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall based on exact matches between predicted and gold keyphrases, types, and relations. Micro-averaged across publications of the three genres (Computer Science, Material Sciences, Physics).

Input / output format

Input: Plain text of scientific publications, optionally accompanied by manually annotated keyphrase boundaries and/or keyphrase types depending on the evaluation scenario.

Output: A list of extracted keyphrase mentions, their assigned types (PROCESS, TASK, or MATERIAL), and the semantic relations (HYPONYM-OF or SYNONYM-OF) linking them.

Scoring recipe

def compute_metrics(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
# Final score is micro-averaged across all publications in a genre

Common pitfalls

• Evaluators must use exact match criteria rather than semantic similarity metrics, as explicitly stated in the protocol.
• The final reported score must be micro-averaged across publications within each of the three scientific genres, not macro-averaged or averaged globally without genre stratification.

Evidence (verbatim from paper)

The traditionally used metrics of precision, recall and F1-score are computed and the micro-average of those metrics across publications of the three genres are calculated. These metrics are also calculated for Subtasks B and C.

Citation

@misc{augenstein2017scienceie,
  title={SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications},
  author={Augenstein et al. (2017)},
  year={2017},
  note={SemEval 2017}
}

• arXiv: 1704.02853