# Llms4ol2024 Eval

> Evaluates LLMs on ontology learning tasks including term typing, taxonomy induction, and non-taxonomic relation extraction across multiple domains and few-shot/zero-shot settings. Use when the user wants to benchmark on LLMs4OL-2024, or asks about evaluating this task. Reports F1-score.

- Skill: `qhjqhj00/llms4ol2024-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/llms4ol2024-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/llms4ol2024-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/llms4ol2024-eval

---


# llms4ol2024-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Evaluates LLMs on ontology learning tasks including term typing, taxonomy induction, and non-taxonomic relation extraction across multiple domains and few-shot/zero-shot settings.

## Datasets

- **LLMs4OL-2024** — total ?; splits: train (-1), test (-1); repo https://codalab.lisn.upsaclay.fr/competitions/19547

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Standard F1 score for extracting terms, typed relations, and induced taxonomies from unstructured text.

## Input / output format

**Input**: Unstructured text or ontology seed data.

**Output**: Extracted terms, typed relations, and induced taxonomies.

## Scoring recipe

```python
pred_ontology = model.extract_ontology(text)
gold_ontology = gold_annotations
f1 = f1_score(gold_ontology, pred_ontology)
```

## Common pitfalls

- The challenge comprises three tasks and 21 subtasks, making it difficult to compare overall performance without normalizing across subtasks.
- Evaluation covers both few-shot and zero-shot setups, which require careful handling of prompt templates and context windows to avoid metric inflation.

## Evidence (verbatim from paper)

> The challenge comprised three tasks and 21 subtasks, using datasets derived from ontologies such as WordNet, GeoNames, and UMLS. Datasets were annotated for both few-shot and zero-shot setups, allowing for robust evaluation of model generalisation.

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141