Somd2025 Eval

somd2025-eval

NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

What this evaluates

Probes the capability of joint entity and relation extraction for identifying software mentions and their attributes (URLs, versions, licenses) in scholarly articles. It specifically tests in-distribution performance and out-of-distribution generalization across two competition phases.

Datasets

• SOMD 2025 — total ?; splits: train (-1), test (-1); repo https://sdproc.org/2025/somd25.html

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, averaged across the entity extraction and relation classification subtasks.

Input / output format

Input: Full text of scholarly articles containing software mentions.

Output: Extracted software entities, their associated attributes, and classified relations between them.

Scoring recipe

pred_entities, pred_relations = model.predict(text)
gold_entities, gold_relations = gold_labels
f1_entity = f1_score(gold_entities, pred_entities)
f1_relation = f1_score(gold_relations, pred_relations)
avg_f1 = (f1_entity + f1_relation) / 2

Common pitfalls

• Phase I uses a gold standard training set while Phase II tests out-of-distribution generalization, causing significant performance drops that are often conflated.
• The metric averages entity extraction and relation classification F1 scores, which can mask poor performance on one subtask.

Evidence (verbatim from paper)

Phase I involved model development using a gold standard training set, evaluated using the average score of the entities extraction and relation classification tasks, while phase II tested generalisability on an out-of-distribution set. The competition (March/April 2025) attracted six teams, the best one achieved F1 scores of 0.89 (Phase I) and 0.63 (Phase II), surpassing baselines of 0.804 and 0.491, respectively.

Citation

@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}

• arXiv: 2509.22141