Cochrane Rct Summarization Eval

cochrane-rct-summarization-eval

Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization — Wallace et al. (2020) (arXiv:2008.11293, 2020)

What this evaluates

Evaluates neural abstractive summarization models on their ability to generate factual, fluent, and relevant narrative summaries of randomized controlled trials (RCTs) from Cochrane systematic reviews. Probes the models' susceptibility to hallucination and their capacity to correctly infer the directionality of clinical findings.

Datasets

• Cochrane RCT Summaries — total 100; splits: test (100); repo https://github.com/bwallace/RCT-summarization-data

Metrics

• ROUGE-L — range: [0, 1]
  • Standard ROUGE-L n-gram overlap score between the generated summary and the manually composed reference summary.
• Manual Factuality (primary) — range: 1-5
  • Human annotator rates the degree of agreement between the generated summary and the reference summary's key conclusions on a 5-point Likert scale (1-5).
• findings-JSD — range: [0, 1]
  • Jensen-Shannon Divergence between the probability distributions of clinical findings (e.g., beneficial, harmful, neutral) extracted from the reference and generated summaries.

Input / output format

Input: Set of input documents (typically abstracts or findings sections from multiple RCTs within a Cochrane review).

Output: A single narrative summary text synthesizing the findings of the input RCTs.

Scoring recipe

def compute_rouge_l(generated, reference):
    return rouge_score(reference, generated, rouge_types=['rougeL'])['rougeL.fmeasure']

def compute_findings_jsd(generated, reference):
    p_ref = extract_finding_distribution(reference)
    p_gen = extract_finding_distribution(generated)
    return jensen_shannon_divergence(p_ref, p_gen)

# Manual Factuality requires human annotation:
# 1. Annotator reads generated summary and reference summary.
# 2. Annotator rates agreement on key conclusions/directionality on a 1-5 Likert scale.

Common pitfalls

• ROUGE-L scores are high and comparable across systems but do not correlate with factual accuracy.
• Manual factuality annotation is highly subjective and requires domain expertise; inter-annotator agreement can be low.
• The proposed findings-JSD metric correlates with human factuality but has low predictive power (R^2=0.05) for distinguishing model variants.

Evidence (verbatim from paper)

We report ROUGE-L scores with respect to the target (manually composed) Cochrane summaries, for both the development and test sets in Table 2. The methods perform about comparably with respect to this automatic metric. But ROUGE measures are based on (exact) n-gram overlap, and are insufficient for measuring the factuality of generated texts. Indeed, we find that the summaries generated by all variants considered enjoy strong fluency, but the key question for this application is whether generated summaries are factually correct.

Citation

@misc{wallace2020generating,
  title={Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization},
  author={Wallace et al. (2020)},
  year={2020},
  note={arXiv:2008.11293}
}

• arXiv: 2008.11293