Llms4subjects Eval

llms4subjects-eval

SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog — D'Souza et al. (2025) (arXiv:2504.07199, 2025)

What this evaluates

Evaluates LLMs' capability to perform multilingual subject tagging for technical library records by ranking relevant GND taxonomy subjects based on title and abstract. It probes the model's ability to handle large-scale taxonomies, bilingual semantic processing, and customizable top-k ranking for digital library classification.

Datasets

• all-subjects — total 95603; splits: train (81937), dev (13666); repo https://github.com/jd-coderepos/llms4subjects/tree/main/shared-task-datasets/TIBKAT/all-subjects
• tib-core — total 48882; splits: train (41902), dev (6980); repo https://github.com/jd-coderepos/llms4subjects/tree/main/shared-task-datasets/TIBKAT/tib-core-subjects

Metrics

• top-k ranked list (primary) — range: other
  • Not explicitly defined in the provided text; evaluates the quality of a customizable top-k ranked list of GND subjects against gold annotations. Standard ranking metrics (e.g., MRR, NDCG@k, Recall@k) are typically used for this task.

Input / output format

Input: Record title and abstract (English or German).

Output: Customizable top-k ranked list of relevant GND subject headings.

Scoring recipe

def evaluate(predictions, gold_subjects, k):
    # predictions: list of subject IDs ranked by model
    # gold_subjects: set of correct GND subject IDs
    ranked = predictions[:k]
    hits = sum(1 for sub in ranked if sub in gold_subjects)
    return hits / len(gold_subjects)  # Recall@k
# Note: Exact metric formula not specified in text; shared tasks typically use MRR/NDCG/Recall@k.

Common pitfalls

• Taxonomy size is large (up to 204,739 subjects), requiring efficient retrieval or filtering strategies.
• Task requires bilingual (English/German) semantic processing, which may cause performance drops if models are monolingual.
• Customizable top-k means evaluation must be conducted at multiple k values rather than a fixed cutoff.

Evidence (verbatim from paper)

Given a record’s title and abstract as input, systems were expected to generate a customizable top-k ranked list of relevant GND subjects.

Citation

@misc{dsouza2025llms4subjects,
  title={SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog},
  author={D'Souza et al. (2025)},
  year={2025},
  note={arXiv:2504.07199}
}

• arXiv: 2504.07199