Scicm Scieval

scicm-scieval

All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction — Li et al. (2023) (arXiv:2311.08189, 2023)

What this evaluates

Evaluates cross-modality scientific information extraction by jointly predicting named entities, result entities, and relations from both full-text paragraphs and scientific tables. It probes a model's ability to handle long documents, align entities across modalities, and generalize across different scientific domains.

Datasets

• ScICM — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of precision and recall: F1 = 2 * (P * R) / (P + R). Precision and recall are computed based on exact match of entity boundaries and types for NER, and exact match of subject/object boundaries and relation types for RE.

Input / output format

Input: Scientific papers containing full-text paragraphs and embedded tables. For NER, input is sentence-level text spans. For RE, input is table structure with cell contents and identified subject/object entity spans.

Output: For NER: predicted entity start/end offsets and entity type labels. For RE: predicted subject entity span, object entity span, and relation type label.

Scoring recipe

def compute_f1(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g and p is not None)
    fn = sum(1 for p, g in zip(preds, golds) if p is None and g is not None)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100

Common pitfalls

• Over-annotation of overly general terms as entities instead of specific scientific terms.
• Missing abbreviations for long entity names during auto-annotation.
• Incorrectly splitting nested entities (e.g., treating 'Bi-LSTM-CRF' and 'CNN-char' separately instead of as one).
• Table structure inconsistencies causing entity type errors or misleading relation extraction.

Evidence (verbatim from paper)

We follow the standard evaluation protocol (Zhong and Chen, 2021) and use precision, recall, and F1 as evaluation metrics. For text NER and table NER, both entity boundary and type are required to be correctly predicted. For table RE, the boundaries of the subject entity and the object entity should be correctly identified.

Citation

@misc{li2023scicm,
  title={All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2311.08189}
}

• arXiv: 2311.08189