Postersum Eval

postersum-eval

PosterSum: A Multimodal Benchmark for Scientific Poster Summarization — Saxena et al. (2025) (arXiv:2502.17540, 2025)

What this evaluates

Evaluates multimodal large language models' ability to generate accurate, abstractive summaries from complex, visually dense scientific posters. It probes layout understanding, visual-textual integration, and hierarchical summarization capabilities by measuring how well models extract and synthesize information from combined image and text inputs.

Datasets

• PosterSum — total 16305; splits: test (-1); repo https://github.com/saxenarohit/postersum

Metrics

• ROUGE-L (primary) — range: percent
  • Computes the longest common subsequence between the reference abstract and the generated summary, normalized by the reference length. Standard NLTK or rouge-score implementation is used.
• ROUGE-1 — range: percent
  • Measures unigram overlap between reference and generated text, normalized by reference length. Standard implementation.
• ROUGE-2 — range: percent
  • Measures bigram overlap between reference and generated text, normalized by reference length. Standard implementation.
• METEOR — range: percent
  • Matches generated text to reference using exact, stem, synonym, and paraphrase matches, then applies a penalty for fragmentation. Standard implementation.
• BERTScore-F1 — range: percent
  • Computes token-level similarity using contextual embeddings from a pre-trained BERT model, then calculates precision, recall, and F1 score.
• SacreBLEU — range: percent
  • Corpus-level BLEU score with standard tokenization and sentence splitting (paper contains a typo 'ScareBLEU'). Standard implementation.

Input / output format

Input: Multimodal scientific poster image (containing text, figures, charts, and layout) paired with its corresponding ground-truth abstract for reference.

Output: A single abstractive summary text (poster abstract) generated by the model.

Scoring recipe

def compute_metrics(predictions, references):
    rouge_l = rouge_score(references, predictions, rouge_types=['l'])['f'] * 100
    rouge_1 = rouge_score(references, predictions, rouge_types=['r'])['f'] * 100
    rouge_2 = rouge_score(references, predictions, rouge_types=['2'])['f'] * 100
    meteor = meteor_score(references, predictions) * 100
    bertscore = bert_score.score(predictions, references, lang='en')[2].mean() * 100
    sacrebleu = sacrebleu.corpus_sacrebleu(predictions, [references]).score
    return {'ROUGE-L': rouge_l, 'ROUGE-1': rouge_1, 'ROUGE-2': rouge_2, 'METEOR': meteor, 'BERTScore-F1': bertscore, 'SacreBLEU': sacrebleu}

Common pitfalls

• Models often fail to extract text correctly from complex layouts, causing pure OCR baselines to underperform naive multimodal models that ignore visual structure.
• Single-pass generation on the full poster image leads to information loss; the benchmark demonstrates that hierarchical, region-segmented approaches significantly outperform end-to-end generation.
• ROUGE-based metrics heavily penalize paraphrasing, so models that closely mirror the original abstract's phrasing score higher than those that produce semantically equivalent but lexically different summaries.

Evidence (verbatim from paper)

Our proposed method outperforms all other models, including closed-source models, on all metrics, achieving ROUGE-1/2/L scores of 46.68, 15.73, and 24.18, respectively, with a 3.14% gain on ROUGE-L compared to open-source models. It also attains a substantially higher ScareBLEU score (12.63) and a BERTScore-F1 of 61.37.

Citation

@misc{saxena2025postersum,
  title={PosterSum: A Multimodal Benchmark for Scientific Poster Summarization},
  author={Saxena et al. (2025)},
  year={2025},
  note={arXiv:2502.17540}
}

• arXiv: 2502.17540