Mediasum Eval

mediasum-eval

MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization — Zhu et al. (2021) (arXiv:2103.06410, 2021)

What this evaluates

Evaluates abstractive dialogue summarization models on long-form, multi-speaker media interviews. It measures how well models can condense multi-topic conversations into concise summaries, and assesses transfer learning capabilities to other dialogue domains.

Datasets

• MediaSum — total 463596; splits: train (443596), val (10000), test (10000)

Metrics

• ROUGE-1, ROUGE-2, ROUGE-L F1 (primary) — range: percent
  • Computes the F1 score of overlapping unigrams (ROUGE-1), bigrams (ROUGE-2), and longest common subsequence (ROUGE-L) between the generated summary and the reference summary. Scores are reported as percentages. Hyperparameters are tuned based on the highest ROUGE-L score on the validation set.

Input / output format

Input: Concatenated transcript of all dialogue turns, with each turn prepended by the speaker name.

Output: Abstractive summary of the dialogue.

Scoring recipe

def compute_rouge_f1(predictions, references):
    scores = {}
    for rouge_type in ['rouge1', 'rouge2', 'rougeL']:
        f1_scores = [rouge_score(ref, pred, use_stemmer=True, rouge_types=[rouge_type]).fmeasure
                     for pred, ref in zip(predictions, references)]
        scores[f'ROUGE-{rouge_type[5:].upper()}'] = sum(f1_scores) / len(f1_scores) * 100
    return scores

Common pitfalls

• MediaSum summaries are extremely short (avg 14.4 words), causing standard summarization models to underperform if not tuned for extreme compression.
• The dataset exhibits positional bias (TV interviews place key terms early, radio distributes them), so extractive baselines like LEAD-3 perform poorly (~15 ROUGE).
• Hyperparameter selection is explicitly tied to ROUGE-L on the validation set, not ROUGE-1 or ROUGE-2, which can skew model comparison if ignored.

Evidence (verbatim from paper)

The input concatenates transcripts from all turns, each prepended with the speaker name. We randomly select 10K instances for validation and another 10K for test. We use the ROUGE (Lin, 2004) metrics and hyper-parameters are chosen based on the highest ROUGE-L score on the validation set.

Citation

@misc{zhu2021mediasum,
  title={MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization},
  author={Zhu et al. (2021)},
  year={2021},
  note={arXiv:2103.06410}
}

• arXiv: 2103.06410