Sofc Exp Eval

sofc-exp-eval

The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain — Friedrich et al. (2020) (arXiv:2006.03039, 2020)

What this evaluates

Evaluates neural models on three information extraction sub-tasks in materials science: detecting experiment-describing sentences, extracting and typing entity mentions (materials, values, devices), and filling experiment-specific slots (e.g., temperature, anode material).

Datasets

• SOFC-Exp Corpus — total ?; splits: train (-1), dev (-1), test (-1)
• Synthesis Procedures Dataset — total 230; splits: train (200), dev (15), test (15)

Metrics

• macro-average F1 (primary) — range: [0, 1]
  • Standard F1 score computed as the harmonic mean of precision and recall. Reported as a macro-average across all classes or slots.

Input / output format

Input: Tokenized sentences or documents from materials science publications, processed with ChemDataExtractor for chemical formulas and units.

Output: Binary label (experiment vs. non-experiment) for sentence detection; sequence tags (BIO format) for entity mention detection and slot filling.

Scoring recipe

def compute_macro_f1(predictions, gold):
    precisions, recalls, f1s = [], [], []
    for class_label in unique_classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == class_label and g == class_label)
        fp = sum(1 for p, g in zip(predictions, gold) if p == class_label and g != class_label)
        fn = sum(1 for p, g in zip(predictions, gold) if p != class_label and g == class_label)
        p = tp / (tp + fp) if (tp + fp) > 0 else 0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        precisions.append(p)
        recalls.append(r)
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• Training sets for sentence detection downsample non-experiment sentences by 0.3, which may inflate test performance if not accounted for.
• Inter-annotator agreement statistics are calculated on a data subset and are explicitly noted as not directly comparable to model scores.
• Entity typing for MATERIAL and DEVICE is significantly harder than VALUE due to rare words and lack of explicit discourse cues.

Evidence (verbatim from paper)

We tune our models in a 5-fold cross-validation setting. We also report the mean and standard deviation across those folds as development results. For the test set, we report the macro-average of the scores obtained when applying each of the five models to the test set. Table 7 shows the macro-average F1 scores for our different models on the slot identification task.

Citation

@misc{friedrich2020sofcexp,
  title={The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain},
  author={Friedrich et al. (2020)},
  year={2020},
  note={arXiv:2006.03039}
}

• arXiv: 2006.03039