Domainsum Eval

domainsum-eval

DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization — Yuan et al. (2024) (arXiv:2410.15687, 2024)

What this evaluates

Evaluates abstractive text summarization models under varying degrees of domain shift (genre, style, topic) to measure performance degradation and distributional divergence across hierarchical granularity levels.

Datasets

• DomainSum — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/hpzhang94/DomainSum

Metrics

• ROUGE (primary) — range: percent
  • Measures n-gram overlap between generated and reference summaries. Typically computed for ROUGE-1, ROUGE-2, and ROUGE-L.
• BERTScore — range: [0, 1]
  • Computes cosine similarity between contextual embeddings of generated and reference tokens, aggregated across precision, recall, and F1.

Input / output format

Input: Source document text; optionally, two-shot prompting examples (document-summary pairs).

Output: Generated summary text.

Scoring recipe

def compute_metrics(predictions, references):
    rouge_scores = rouge.compute(predictions=predictions, references=references)
    bert_scores = bertscore.compute(predictions=predictions, references=references)
    return {'ROUGE': rouge_scores, 'BERTScore': bert_scores}

Common pitfalls

• Performance varies significantly between zero-shot and two-shot prompting, with two-shot sometimes degrading performance on certain domains.
• Domain shift granularity (genre vs. style vs. topic) affects metrics non-uniformly; topic shifts show the most consistent distributional change but not necessarily the worst ROUGE scores.
• Models may overfit to in-domain data, making cross-domain evaluation sensitive to training data composition.

Evidence (verbatim from paper)

ROUGE | BERTScore | ROUGE | BERTScore | ROUGE | BERTScore | ROUGE | BERTScore | ROUGE | BERTScore | Table 2: Results for zero-shot and two-shot prompting across genre, style, and topic shift levels in DomainSum. Values in parentheses indicate the performance difference between two-shot and zero-shot prompting. The best results for zero-shot prompting are highlighted in red, two-shot in blue, and cases where both achieve the best results are highlighted in purple.

Citation

@misc{yuan2024domainsum,
  title={DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization},
  author={Yuan et al. (2024)},
  year={2024},
  note={arXiv:2410.15687}
}

• arXiv: 2410.15687