Evidence Inference Eval

evidence-inference-eval

Evidence Inference 2.0: More Data, Better Models — DeYoung et al. (2020) (arXiv:2005.04177, 2020)

What this evaluates

This benchmark evaluates a model's ability to identify and classify clinical evidence spans within randomized controlled trial (RCT) documents. Specifically, it probes whether a given evidence span supports a significantly decreased, no significant difference, or significantly increased outcome relative to a clinical intervention.

Datasets

• Evidence Inference — total ?; splits: test (-1)

Metrics

• macro-averaged F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across three classes: significantly decreased, no significant difference, and significantly increased. Calculated as the unweighted mean of per-class F1 scores.

Input / output format

Input: A clinical intervention/comparison/outcome (ICO) prompt and a candidate evidence span or sentence extracted from a full RCT document or abstract.

Output: One of three classification labels: 'significantly decreased', 'no significant difference', or 'significantly increased'.

Scoring recipe

def compute_macro_f1(predictions, gold):
    classes = ['sig_decreased', 'sig_no_diff', 'sig_increased']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Models must be conditioned on the ICO prompt; unconditioned models drop ~27 F1 points.
• The evaluation uses a two-stage pipeline (evidence identification followed by classification); end-to-end scores are lower than oracle evidence baselines by ~0.08 F1.
• Abstract-only subsets yield nearly identical performance to full-document subsets, but oracle abstract models underperform oracle full-document models due to differences in reporting language.

Evidence (verbatim from paper)

Our main results are reported in Table 1. We make a few key observations. First, the gains over the prior state-of-the-art model — which was not BERT based — are substantial: 20+ absolute points in F-score, even beyond what one might expect to see shifting to large pre-trained models. To verify the impact of architecture changes, we experiment with randomly initialized and fine-tuned BERTs. We find that these perform worse than the original models in all instances and elide more detailed results. Second, conditioning on the ICO prompt is key; failing to do so results in substantial performance drops. Finally, we seem to have reached a plateau in terms of the performance of the BERT pipeline model; adding the newly collected training data does not budge performance (evaluated on the augmented test set). This suggests that to realize stronger performance here, we perhaps need a less naive architecture that better models the domain.

Citation

@misc{deyoung2020evidenceinference2,
  title={Evidence Inference 2.0: More Data, Better Models},
  author={DeYoung et al. (2020)},
  year={2020},
  note={arXiv:2005.04177}
}

• arXiv: 2005.04177