Forc2025 Eval

forc2025-eval

NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

What this evaluates

Evaluates hierarchical multi-label classification of academic papers into a taxonomy of 170 research fields. It tests zero-shot/few-shot prompting and weakly-labeled data integration for field-of-research prediction.

Datasets

• FoRC4CL 2025 — total 42500; splits: train (-1), test (-1); repo https://nfdi4ds.github.io/nslp2025/docs/forc_shared_task.html

Metrics

• Micro-F1 (primary) — range: [0, 1]
  • F1 score calculated globally by counting total true positives, false negatives, and false positives across all 170 labels.

Input / output format

Input: ACL Anthology article text.

Output: Set of predicted labels from a taxonomy of 170 topics.

Scoring recipe

pred_labels = model.predict_labels(article_text)
gold_labels = gold_taxonomy_labels
micro_f1 = f1_score(gold_labels, pred_labels, average='micro')
macro_f1 = f1_score(gold_labels, pred_labels, average='macro')
weighted_f1 = f1_score(gold_labels, pred_labels, average='weighted')

Common pitfalls

• The task uses a large weakly-labeled dataset alongside a small gold-standard set, which may lead to overfitting on noisy labels if not handled carefully.
• Multiple F1 variants (micro, macro, weighted) are reported; micro-F1 is typically primary for imbalanced multi-label tasks but can mask poor performance on rare fields.

Evidence (verbatim from paper)

Their model yielded a micro-F1 of 0.68, a macro-F1 of 0.66, and a weighted-F1 of 0.69, showing the potential of hybrid retrieval-generation pipelines.

Citation

@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}

• arXiv: 2509.22141