Multidomain Summarization Eval

multidomain-summarization-eval

Analysis of Multidomain Abstractive Summarization Using Salience Allocation — Rehman et al. (2024) (arXiv:2402.11955, 2024)

What this evaluates

Evaluates the ability of abstractive summarization models to generate coherent, factually accurate, and semantically aligned summaries across general news, conversational, and financial domains. It probes content selection, hallucination reduction, and domain-specific adaptation by leveraging sentence-level salience signals during generation.

Datasets

• CNN/Dailymail — total ?; splits: train (5000), val (625), test (625)
• SAMSum — total 16000; splits: train (5000), val (625), test (625)
• Financial-news based Event-Driven Trading (EDT) — total ?; splits: train (5000), val (625), test (625)

Metrics

• ROUGE-Lsum (primary) — range: [0, 1]
  • Measures the longest common subsequence between predicted and reference summaries, computed at the sentence level (Lsum) to handle multi-sentence outputs and assess lexical overlap.
• ROUGE-N — range: [0, 1]
  • Computes n-gram overlap (typically unigram to 4-gram) between predicted and reference summaries to assess exact lexical matching.
• METEOR — range: [0, 1]
  • Calculates a score blending unigram precision, recall, and a fragmentation penalty to reflect word ordering alignment and synonymy.
• BERTScore — range: [0, 1]
  • Uses contextual BERT embeddings to compute cosine similarity between predicted and reference tokens, capturing semantic and contextual alignment beyond exact n-grams.
• MoverScore — range: [0, 1]
  • Measures text deviation from references using optimal transport on contextual embeddings, emphasizing faithfulness and semantic distance.

Input / output format

Input: Source text (news articles or chat dialogues), truncated to 512, 256, or 512 tokens respectively per dataset.

Output: Generated summary, truncated to 100, 50, or 40 tokens respectively per dataset. Produced via beam search (width=5), length penalty=1.5, 3-gram blocking, and temperature=0.5 for salience allocation.

Scoring recipe

def compute_metrics(predictions, references):
    # ROUGE-Lsum & ROUGE-N
    rouge = rouge_scorer.RougeScorer(['rougeLsum', 'rouge1', 'rouge2'], use_stemmer=True)
    rouge_scores = {k: rouge.score(ref, pred)[k].fmeasure for k in ['rougeLsum', 'rouge1', 'rouge2']}
    
    # METEOR
    meteor_scores = [meteor_score_fn([ref], [pred]) for ref, pred in zip(references, predictions)]
    
    # BERTScore
    bert_scores = bert_score.compute(predictions, references, lang='en')
    
    # MoverScore
    mover_scores = mover_score.compute(predictions, references)
    
    return {
        'ROUGE-Lsum': rouge_scores['rougeLsum'],
        'ROUGE-N': rouge_scores['rouge1'],
        'METEOR': sum(meteor_scores) / len(meteor_scores),
        'BERTScore': bert_scores['F1'].mean().item(),
        'MoverScore': mover_scores['F1'].mean().item()
    }

Common pitfalls

• Using the full dataset splits instead of the reported 5,000/625/625 subset, which drastically changes evaluation scale and results.
• Ignoring the specific inference hyperparameters (beam size 5, length penalty 1.5, 3-gram blocking) which significantly affect generation quality and lexical overlap.
• Confusing the training-time salience measurement metric (ROUGE-L F1) with the official evaluation metrics reported in the paper.

Evidence (verbatim from paper)

Due to limitations in computational resources, a subset of 5,000 training, 625 validation, and 625 testing data points was employed from each datasets. ... Following the summarization process by fine-tuned of pre-trained models, the generated summaries undergo evaluation using ROUGE, METEOR, BERTScore, and MoverScore metrics. ROUGE [6] metric evaluate the matched model predicted summary to human written summary, measuring n-gram overlap (ROUGE-N) and longest common sequence (ROUGE-L). ROUGE-Lsum offers detailed insights for enhancing NLP summarization task. METEOR metric [7] evaluates the match between model predicted summary to human written summary by assigning a score that considers a blend of unigram precision, unigram recall, and a fragmentation measure.

Citation

@misc{rehman2024analysis,
  title={Analysis of Multidomain Abstractive Summarization Using Salience Allocation},
  author={Rehman et al. (2024)},
  year={2024},
  note={arXiv:2402.11955}
}

• arXiv: 2402.11955