Llms4ol2024 Eval

llms4ol2024-eval

NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

What this evaluates

Evaluates LLMs on ontology learning tasks including term typing, taxonomy induction, and non-taxonomic relation extraction across multiple domains and few-shot/zero-shot settings.

Datasets

• LLMs4OL-2024 — total ?; splits: train (-1), test (-1); repo https://codalab.lisn.upsaclay.fr/competitions/19547

Metrics

• F1-score (primary) — range: [0, 1]
  • Standard F1 score for extracting terms, typed relations, and induced taxonomies from unstructured text.

Input / output format

Input: Unstructured text or ontology seed data.

Output: Extracted terms, typed relations, and induced taxonomies.

Scoring recipe

pred_ontology = model.extract_ontology(text)
gold_ontology = gold_annotations
f1 = f1_score(gold_ontology, pred_ontology)

Common pitfalls

• The challenge comprises three tasks and 21 subtasks, making it difficult to compare overall performance without normalizing across subtasks.
• Evaluation covers both few-shot and zero-shot setups, which require careful handling of prompt templates and context windows to avoid metric inflation.

Evidence (verbatim from paper)

The challenge comprised three tasks and 21 subtasks, using datasets derived from ontologies such as WordNet, GeoNames, and UMLS. Datasets were annotated for both few-shot and zero-shot setups, allowing for robust evaluation of model generalisation.

Citation

@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}

• arXiv: 2509.22141