Ontology Alignment Eval

ontology-alignment-eval

The Science Data Lake: A Unified Open Infrastructure Integrating 293 Million Papers Across Eight Scholarly Sources with Embedding-Based Ontology Alignment — Wilinski (2026) (arXiv:2603.03126, 2026)

What this evaluates

This benchmark evaluates the accuracy of mapping OpenAlex paper topics to terms across 13 scientific ontologies. It probes a method's ability to perform semantic and lexical alignment between informal topic labels and formal domain-specific vocabularies.

Datasets

• Ontology Alignment Gold Standard — total 300; splits: test (300)

Metrics

• F1 (primary) — range: [0, 1]
  • F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed from unrounded Precision and Recall values.
• Precision — range: [0, 1]
  • Proportion of predicted mappings that are correct in the gold standard.
• Recall — range: [0, 1]
  • Proportion of gold standard mappings that are correctly predicted.

Input / output format

Input: OpenAlex topic labels (4,516 total) and ontology term labels (1.3M total across 13 ontologies).

Output: A single mapped ontology term for each OpenAlex topic, or null if no mapping is produced.

Scoring recipe

def compute_metrics(predictions, gold):
    correct = sum(1 for t, pred in predictions.items() if (t, pred) in gold)
    total_preds = len(predictions)
    total_gold = len(gold)
    precision = correct / total_preds if total_preds > 0 else 0.0
    recall = correct / total_gold if total_gold > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• "Partial" matches in the gold standard are strictly counted as false positives, not partial credit.
• F1 must be computed from unrounded Precision and Recall values; rounding them first yields slightly different F1 scores.
• Large ontologies (MeSH, ChEBI, NCIT) bypass the embedding search and use exact string matching, which affects the overall alignment distribution.

Evidence (verbatim from paper)

Precision (P), recall (R), and F1 are evaluated against a stratified 300-pair gold-standard annotation set (see Section 4); a mapping is counted as correct only if the annotator labelled it correct (strict; partial counts as false positive). F1 is computed from unrounded P and R; rounding the displayed values may yield slightly different results.

Citation

@misc{wilinski2026sciencedatalake,
  title={The Science Data Lake: A Unified Open Infrastructure Integrating 293 Million Papers Across Eight Scholarly Sources with Embedding-Based Ontology Alignment},
  author={Wilinski (2026)},
  year={2026},
  note={arXiv:2603.03126}
}

• arXiv: 2603.03126