Symlink Eval

symlink-eval

SemEval 2022 Task 12: Symlink- Linking Mathematical Symbols to their Descriptions — Lai et al. (2022) (arXiv:2202.09695, 2022)

What this evaluates

Probes the ability to extract fine-grained mathematical symbols and their textual descriptions from LaTeX-formatted scientific documents. It evaluates both named entity recognition for identifying symbols and descriptions, and relation extraction for linking them according to specific semantic types.

Datasets

• Symlink — total ?; splits: test (-1); repo https://github.com/nlp-uoregon/symlink

Metrics

• F1 (partial) — range: percent
  • Entity-based partial F1 score following SemEval 2013 Task 9.1 guidelines, measuring span overlap without requiring exact type matching.
• F1 (type) — range: percent
  • Entity-based type F1 score following SemEval 2013 Task 9.1 guidelines, requiring exact span and type matching.
• F-score (primary) — range: percent
  • Standard harmonic mean of precision and recall for relation extraction. A relation prediction is correct only if the label strictly matches the gold standard.

Input / output format

Input: LaTeX source text from scientific documents containing mathematical symbols and their surrounding descriptions.

Output: Predicted entity labels for NER (symbol/description) and relation labels (Coref-Description, Coref-Symbol, Direct, Count) that must strictly match the gold standard.

Scoring recipe

def evaluate_ner(pred_entities, gold_entities):
    # Calculate partial and type-based F1 per SemEval 2013 Task 9.1
    partial_f1 = compute_f1(pred_entities, gold_entities, type_match=False)
    type_f1 = compute_f1(pred_entities, gold_entities, type_match=True)
    return partial_f1, type_f1

def evaluate_re(pred_relations, gold_relations):
    tp = sum(1 for p in pred_relations if p in gold_relations)
    fp = len(pred_relations) - tp
    fn = len(gold_relations) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• Relation predictions require strict label matching; partial or fuzzy matches are counted as incorrect.
• NER evaluation separates partial span matches from exact type matches, which can yield different F1 scores.
• Systems may choose to submit only for the NER or RE subtask, so overall leaderboard ranking is based on RE F-score.

Evidence (verbatim from paper)

For NER, we use the entity-based partial/type from SemEval 2013 Task 9.1. For RE, we use standard precision, recall, F-score metrics. Relations output by the participating system is correct if the prediction label strictly matches the gold standard.

Citation

@misc{lai2022symlink,
  title={SemEval 2022 Task 12: Symlink- Linking Mathematical Symbols to their Descriptions},
  author={Lai et al. (2022)},
  year={2022},
  note={arXiv:2202.09695}
}

• arXiv: 2202.09695