Matsci Nlp Eval

matsci-nlp-eval

MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling — Yu Song, Santiago Miret, Bang Liu (2023) (arXiv:2305.08264, 2023)

What this evaluates

Evaluates scientific language models on seven materials science NLP tasks, including named entity recognition, relation classification, event argument extraction, paragraph classification, synthesis action retrieval, sentence classification, and slot filling. It probes the model's ability to extract structured information and classify text from domain-specific scientific literature.

Datasets

• MatSci-NLP — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• micro-F1 (primary) — range: [0, 1]
  • Weighted average of F1 scores across all classes, where each sample's contribution is weighted by its class frequency.
• macro-F1 — range: [0, 1]
  • Unweighted mean of F1 scores across all classes, giving equal weight to each class regardless of frequency.

Input / output format

Input: Raw text excerpts from materials science literature (e.g., abstracts, paragraphs, sentences).

Output: Task-specific labels: token-level entity tags for NER, relation labels for relation classification, event argument roles for event extraction, paragraph/sentence class labels, retrieved synthesis actions, and slot values.

Scoring recipe

For each task, compute per-class precision, recall, and F1.
micro-F1 = average of per-sample F1 scores weighted by class frequency.
macro-F1 = unweighted mean of per-class F1 scores.
Report mean and ±2 standard deviations across 5 independent runs.

Common pitfalls

• Results are reported as mean ± 2 standard deviations across 5 independent runs, not a single train/val/test split.
• Both micro-F1 and macro-F1 are reported; macro-F1 can be significantly lower for imbalanced datasets.
• Evaluates multiple prompting/fine-tuning strategies (Single Task, Text2Schema, etc.) rather than a single fixed protocol.

Evidence (verbatim from paper)

The results for seven tasks in MatSci-NLP are shown in subsequent tables: Named Entity Recognition in Table 4. Relation Classification in Table 5. Event Argument Extraction in Table 6. Paragraph Classification in Table 7. Synthesis Action Retrieval in Table 8. Sentence Classification in Table 9. Slot Filling in Table 10. For each model, the top line represents the microF1 score and the bottom line represents the macro-F1 score. We report the mean across 5 experiments with a confidence interval of two standard deviations.

Citation

@misc{song2023matscinlp,
  title={MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling},
  author={Yu Song, Santiago Miret, Bang Liu (2023)},
  year={2023},
  note={arXiv:2305.08264}
}

• arXiv: 2305.08264