Graphsum Mds Eval

graphsum-mds-eval

Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization — Hickmann et al. (2021) (arXiv:2105.11908, 2021)

What this evaluates

Evaluates multi-document summarization performance and model explainability by comparing sentence vs. paragraph inputs and analyzing how attention weights correlate with reference summary similarity to reveal positional bias.

Datasets

• MultiNews — total ?; splits: test (-1)
• WikiSum — total ?; splits: test (-1)

Metrics

• ROUGE-F (1/2/L) (primary) — range: percent
  • Standard ROUGE-F1, F2, and F-L scores measuring n-gram overlap between generated and reference summaries, scaled to 0-100.
• correlation coefficient — range: [-1, 1]
  • Pearson correlation coefficient computed between aggregated attention weights per decoding layer and ROUGE scores.

Input / output format

Input: Multiple source documents represented as sequences of tokens or paragraph-level textual units.

Output: Abstractive summary generated token-by-token.

Scoring recipe

# Compute ROUGE-F scores for each generated summary
rouge_scores = compute_rouge_f1_f2_fl(generated_summary, reference_summary)

# Aggregate attention weights per decoding layer
attention_weights = extract_aggregated_attention_weights(model, layer)

# Compute correlation between attention weights and ROUGE scores
correlation = pearson_correlation(attention_weights, rouge_scores)

Common pitfalls

• Positional bias in news summaries heavily influences attention patterns, making later layers appear more correlated with source origin.
• Heatmap visualizations of attention artifacts are skewed by the distribution of summary lengths in the test set.
• WikiSum lacks document/paragraph boundary metadata, preventing positional bias analysis on that dataset.

Evidence (verbatim from paper)

Table 1 shows the results for the comparison of sentence-level vs. paragraph-level summaries generated by our MDS models on the test dataset. With a batchsize of 3,000 target tokens, the paragraph-level model outperforms the sentence-level model with regard to the ROUGE score by a small margin.

Citation

@misc{hickmann2021graphsum,
  title={Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization},
  author={Hickmann et al. (2021)},
  year={2021},
  note={arXiv:2105.11908}
}

• arXiv: 2105.11908