# Symlink Eval

> Probes the ability to extract fine-grained mathematical symbols and their textual descriptions from LaTeX-formatted scientific documents. It evaluates both named entity recognition for identifying symbols and descriptions, and relation extraction for linking them according to specific semantic types. Use when the user wants to benchmark on Symlink, or asks about evaluating this task. Reports F-score.

- Skill: `qhjqhj00/symlink-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/symlink-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/symlink-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/symlink-eval

---


# symlink-eval

> SemEval 2022 Task 12: Symlink- Linking Mathematical Symbols to their Descriptions — Lai et al. (2022) (arXiv:2202.09695, 2022)

## What this evaluates

Probes the ability to extract fine-grained mathematical symbols and their textual descriptions from LaTeX-formatted scientific documents. It evaluates both named entity recognition for identifying symbols and descriptions, and relation extraction for linking them according to specific semantic types.

## Datasets

- **Symlink** — total ?; splits: test (-1); repo https://github.com/nlp-uoregon/symlink

## Metrics

- `F1 (partial)` — range: percent
  - Entity-based partial F1 score following SemEval 2013 Task 9.1 guidelines, measuring span overlap without requiring exact type matching.
- `F1 (type)` — range: percent
  - Entity-based type F1 score following SemEval 2013 Task 9.1 guidelines, requiring exact span and type matching.
- `F-score` **(primary)** — range: percent
  - Standard harmonic mean of precision and recall for relation extraction. A relation prediction is correct only if the label strictly matches the gold standard.

## Input / output format

**Input**: LaTeX source text from scientific documents containing mathematical symbols and their surrounding descriptions.

**Output**: Predicted entity labels for NER (symbol/description) and relation labels (Coref-Description, Coref-Symbol, Direct, Count) that must strictly match the gold standard.

## Scoring recipe

```python
def evaluate_ner(pred_entities, gold_entities):
    # Calculate partial and type-based F1 per SemEval 2013 Task 9.1
    partial_f1 = compute_f1(pred_entities, gold_entities, type_match=False)
    type_f1 = compute_f1(pred_entities, gold_entities, type_match=True)
    return partial_f1, type_f1

def evaluate_re(pred_relations, gold_relations):
    tp = sum(1 for p in pred_relations if p in gold_relations)
    fp = len(pred_relations) - tp
    fn = len(gold_relations) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- Relation predictions require strict label matching; partial or fuzzy matches are counted as incorrect.
- NER evaluation separates partial span matches from exact type matches, which can yield different F1 scores.
- Systems may choose to submit only for the NER or RE subtask, so overall leaderboard ranking is based on RE F-score.

## Evidence (verbatim from paper)

> For NER, we use the entity-based partial/type from SemEval 2013 Task 9.1. For RE, we use standard precision, recall, F-score metrics. Relations output by the participating system is correct if the prediction label strictly matches the gold standard.

## Citation

```bibtex
@misc{lai2022symlink,
  title={SemEval 2022 Task 12: Symlink- Linking Mathematical Symbols to their Descriptions},
  author={Lai et al. (2022)},
  year={2022},
  note={arXiv:2202.09695}
}
```

- arXiv: 2202.09695