Sourcedata Nlp Eval

sourcedata-nlp-eval

Integrating curation into scientific publishing to train AI models — Abreu Vicente et al. (2023) (arXiv:2310.20440, 2023)

What this evaluates

Evaluates biomedical named entity recognition (NER) capabilities on scientific literature. It probes a model's ability to identify and classify nine distinct bioentity types (e.g., genes, cell lines, diseases) within text extracted from published biological figures and captions.

Datasets

• SourceData-NLP — total 801818; splits: train (-1), val (-1), test (-1); repo https://github.com/source-data/soda-data

Metrics

• F1 score (primary) — range: percent
  • Harmonic mean of precision and recall. Calculated per entity class and aggregated via micro, macro, or weighted averaging. Reported as the mean across 5 inference rounds.

Input / output format

Input: Text spans from scientific articles and figure captions containing biological entities.

Output: Span-level labels from a fixed set of nine entity categories (Gene product, Cell line, Organism, Small molecule, Tissue, Subcellular, Cell type, Disease, Exp. Assay) and optional external database identifiers.

Scoring recipe

def compute_f1(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g and p != 'O')
    fn = sum(1 for p, g in zip(preds, golds) if p != g and g != 'O')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Entity classes are highly imbalanced (e.g., gene products vs. diseases), causing macro-averaged F1 to differ significantly from micro-averaged F1.
• Splits are stratified by article/figure rather than randomly shuffled, risking data leakage if related papers share similar terminology or experimental designs.
• The paper also evaluates a separate semantic role classification task and figure segmentation task, which are sometimes conflated with the core NER benchmark.

Evidence (verbatim from paper)

The dataset is segmented into training, evaluation, and test sets in an 80-10-10 ratio. In total, 801,818 entities are provided, including 686,846 entities linked to identifiers in external databases. ... The results show the average F1 scores obtained through 5 inference rounds.

Citation

@misc{abreu2023sourcedatanlp,
  title={Integrating curation into scientific publishing to train AI models},
  author={Abreu Vicente et al. (2023)},
  year={2023},
  note={arXiv:2310.20440}
}

• arXiv: 2310.20440