Spert Eval

spert-eval

Span-based Joint Entity and Relation Extraction with Transformer Pre-training — Eberts et al. (2019) (arXiv:1909.07755, 2019)

What this evaluates

Evaluates a model's ability to jointly identify named entity spans with their types and extract relational tuples between them from unstructured text. It probes span-based representation learning, localized context modeling, and joint classification without relying on sequential tagging schemes like BIO.

Datasets

• CoNLL04 — total ?; splits: train (1153), dev (230), test (288)
• SciERC — total 2687; splits: train (1861), dev (275), test (551)
• ADE — total 4272; splits: 10-fold CV (4272)

Metrics

• F1 score (micro/macro-averaged) (primary) — range: [0, 1]
  • Precision = TP / (TP + FP), Recall = TP / (TP + FN), F1 = 2 * (Precision * Recall) / (Precision + Recall). Entities are correct if predicted span and label match gold. Relations are correct if relation type and both head/tail entities match gold in span and type. For SciERC, entity type is ignored for relation correctness. Micro-averaging is used for SciERC; both micro and macro are reported for CoNLL04 and ADE. ADE scores are averaged over 10 folds.

Input / output format

Input: Raw sentence text.

Output: A set of predicted entity spans with type labels, and a set of predicted relation tuples (head entity span, tail entity span, relation type).

Scoring recipe

def evaluate(pred_entities, gold_entities, pred_relations, gold_relations, ignore_entity_type=False):
    tp_e = len(set(pred_entities) & set(gold_entities))
    prec_e = tp_e / len(pred_entities) if pred_entities else 0
    rec_e = tp_e / len(gold_entities) if gold_entities else 0
    f1_e = 2 * prec_e * rec_e / (prec_e + rec_e) if (prec_e + rec_e) > 0 else 0

    def rel_match(p, g):
        if p.head.span != g.head.span or p.tail.span != g.tail.span: return False
        if not ignore_entity_type and (p.head.type != g.head.type or p.tail.type != g.tail.type): return False
        return p.type == g.type

    tp_r = sum(1 for p in pred_relations if any(rel_match(p, g) for g in gold_relations))
    prec_r = tp_r / len(pred_relations) if pred_relations else 0
    rec_r = tp_r / len(gold_relations) if gold_relations else 0
    f1_r = 2 * prec_r * rec_r / (prec_r + rec_r) if (prec_r + rec_r) > 0 else 0
    return f1_e, f1_r

Common pitfalls

• SciERC relation evaluation explicitly ignores entity type correctness, unlike CoNLL04 and ADE.
• ADE contains 120 relations with overlapping entities that prior benchmarks filtered out; SpERT evaluates on the full unfiltered dataset.
• Micro-averaging is standard for SciERC, while both micro and macro are reported for CoNLL04 and ADE due to inconsistent reporting in prior work.

Evidence (verbatim from paper)

We evaluate SpERT on both entity recognition and relation extraction. An entity is considered correct if its predicted span and entity label match the ground truth. A relation is considered correct if its relation type as well as the two related entities are both correct (in span and type). Only for SciERC, entity type correctness is not considered when evaluating relation extraction, which is in line with prior work [20, 21, 34]. Following previous work, we measure the precision, recall and F1 score for entities and relations, and report micro-averaged values for the SciERC dataset.

Citation

@misc{eberts2019spert,
  title={Span-based Joint Entity and Relation Extraction with Transformer Pre-training},
  author={Eberts et al. (2019)},
  year={2019},
  note={arXiv:1909.07755}
}

• arXiv: 1909.07755