qhjqhj00 pio-detailed-labeling-eval
A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature — Nye et al. (2018) (arXiv:1806.04185, 2018)
What this evaluates
Evaluates a model's ability to predict fine-grained hierarchical labels for tokens within identified P, I, and O spans.
Datasets
- EBM-NLP — total 5000; splits: train (4300), dev (500), test (200)
Metrics
F-1(primary) — range: [0, 1]- Harmonic mean of precision and recall: (2 * P * R) / (P + R). Computed per element type (P, I, O) and averaged across the test set.
Input / output format
Input: Raw clinical trial abstract text.
Output: Token-level hierarchical labels from the PIO hierarchy, predicted only for tokens within identified spans.
Scoring recipe
def compute_f1(predictions, gold):
pred_labels = [p for p in predictions if p != 'O']
gold_labels = [g for g in gold if g != 'O']
tp = sum(1 for p, g in zip(pred_labels, gold_labels) if p == g)
fp = len(pred_labels) - tp
fn = len(gold_labels) - tp
p = tp / (tp + fp) if (tp + fp) > 0 else 0
r = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * p * r / (p + r) if (p + r) > 0 else 0
Common pitfalls
- Initial neural models proved unfruitful, indicating high complexity and potential need for better feature engineering or pre-training.
- Labels are hierarchical; predicting incorrect sub-labels within a span counts as errors.
Evidence (verbatim from paper)
Beyond identifying the spans of text containing information pertinent to each of the PIO elements, we consider the task of predicting which of the detailed labels occur in each span, and where they are located. Specifically, we begin with the starting spans and predict a single label from the corresponding PIO hierarchy for each token, evaluating against the test set of 200 documents. Table 13 reports Precision, Recall, and F-1.
Citation
@misc{nye2018ebmnlp,
title={A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature},
author={Nye et al. (2018)},
year={2018},
note={arXiv:1806.04185}
}
- arXiv: 1806.04185