qhjqhj00 redundancy-detection-eval
A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature — Nye et al. (2018) (arXiv:1806.04185, 2018)
What this evaluates
Evaluates a model's ability to determine whether pairs of P, I, or O spans in an abstract refer to the same underlying information.
Datasets
- EBM-NLP — total 5000; splits: train (4300), dev (500), test (200)
Metrics
F-1(primary) — range: [0, 1]- Harmonic mean of precision and recall: (2 * P * R) / (P + R). Computed for binary redundancy classification across all span pairs.
Input / output format
Input: Pairs of P, I, or O spans extracted from a single abstract.
Output: Binary label (redundant or not) indicating whether the pair shares at least one instance of the same information.
Scoring recipe
def compute_f1(predictions, gold):
tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
p = tp / (tp + fp) if (tp + fp) > 0 else 0
r = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * p * r / (p + r) if (p + r) > 0 else 0
Common pitfalls
- Task is defined over pairs of spans, not single spans, requiring a different input pipeline.
- A large majority of spans contain only a single instance of relevant information, making the task relatively easy as a baseline.
Evidence (verbatim from paper)
To formalize repetition, we consider every pair of starting PIO spans from each abstract, and assign binary labels that indicate whether they share at least one instance of the same information. We train a logistic regression model that operates over standard features, including bag-of-words representations and sentence-level features such as length and position in the document. Table 14 reports Precision, Recall, and F-1.
Citation
@misc{nye2018ebmnlp,
title={A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature},
author={Nye et al. (2018)},
year={2018},
note={arXiv:1806.04185}
}
- arXiv: 1806.04185