Pio Span Tagging Eval

pio-span-tagging-eval

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature — Nye et al. (2018) (arXiv:1806.04185, 2018)

What this evaluates

Evaluates a model's ability to identify text spans corresponding to Patient, Intervention, and Outcome elements within clinical trial abstracts.

Datasets

• EBM-NLP — total 5000; splits: train (4300), dev (500), test (200)

Metrics

• F-1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: (2 * P * R) / (P + R). Computed per element type (P, I, O) and averaged across the test set.

Input / output format

Input: Raw clinical trial abstract text.

Output: Token-level BIO tags (Inside/Outside) indicating whether each token belongs to a Patient, Intervention, or Outcome span.

Scoring recipe

def compute_f1(predictions, gold):
  pred_spans = extract_spans_from_bio(predictions)
  gold_spans = extract_spans_from_bio(gold)
  tp = len(set(pred_spans) & set(gold_spans))
  fp = len(set(pred_spans) - set(gold_spans))
  fn = len(set(gold_spans) - set(pred_spans))
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  return 2 * p * r / (p + r) if (p + r) > 0 else 0

Common pitfalls

• Evaluating on the development set instead of the expert-annotated test set, which compromises reliability.
• Treating the task as sentence-level classification rather than token-level span detection.

Evidence (verbatim from paper)

Note that we have pre-defined train, development and test sets across PIO elements for this corpus, comprising 4300, 500 and 200 abstracts, respectively. The latter set is annotated by domain experts (i.e., persons with medical training). These splits will, of course, be distributed along with the dataset to facilitate model comparisons. We treat tokens as being either Inside (I) or Outside (O) of spans. Table 12 reports Precision, Recall, and F-1 for CRF and LSTM-CRF baselines.

Citation

@misc{nye2018ebmnlp,
  title={A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature},
  author={Nye et al. (2018)},
  year={2018},
  note={arXiv:1806.04185}
}

• arXiv: 1806.04185