qhjqhj00 trec-2020-podcast-summarisation-eval
CUED_speech at TREC 2020 Podcast Summarisation Track — Manakul et al. (2020) (arXiv:2012.02535, 2020)
What this evaluates
Evaluates the ability of summarization systems to generate concise, accurate, and informative summaries of long-form spoken podcast episodes. It probes handling of speech-specific challenges like redundancy, speaker turns, and informal language, as well as factual recall of key entities and events.
Datasets
- TREC 2020 Podcast Summarisation Track — total 179; splits: test (179)
Metrics
Avg(primary) — range: [0, 3]- Average score computed by mapping ordinal human ratings to Bad=0, Fair=1, Good=2, Excellent=3, then averaging across all annotated episodes.
ROUGE-L (F1)— range: percent- Longest common subsequence F1 score between generated summaries and reference summaries. Computed using both official NIST references and a processed version of episode descriptions.
Input / output format
Input: Raw or filtered transcriptions of podcast episodes.
Output: Abstractive text summaries of the podcast episodes.
Scoring recipe
def compute_avg_human(annotations):
score_map = {'Bad': 0, 'Fair': 1, 'Good': 2, 'Excellent': 3}
return sum(score_map[a] for a in annotations) / len(annotations)
def compute_rouge_l_f1(generated, reference):
# Standard ROUGE-L F1 calculation on longest common subsequence
precision = len(lcs(generated, reference)) / len(generated)
recall = len(lcs(generated, reference)) / len(reference)
if precision + recall == 0: return 0.0
return 2 * (precision * recall) / (precision + recall) * 100
Common pitfalls
- Human evaluation uses an ordinal scale (0-3) but is averaged as if interval; results should not be interpreted as percentages or probabilities.
- ROUGE-L scores vary significantly depending on whether official NIST references or processed/cleaned references are used, as spoken transcripts contain noise like URLs and filler words.
- Ensemble size (3 vs 9 models) impacts performance, so single-model results may not match reported best scores.
Evidence (verbatim from paper)
179 episodes were selected randomly, and NIST annotators evaluated them on the Bad/Fair/Good/Excellent scale... Table 6 shows that our CUED-ensemble3 submission receives the highest human rating at 1.777 in average... When using an automatic evaluation, CUED-ensemble3 also achieves the highest score shown in Table 7 (note that our ROUGE scores, computed using processed episode descriptions, suggest the performance of CUED-ensemble3 and CUED-ensemble9 are similar).
Citation
@misc{manakul2020cued_speech,
title={CUED_speech at TREC 2020 Podcast Summarisation Track},
author={Manakul et al. (2020)},
year={2020},
note={arXiv:2012.02535}
}
- arXiv: 2012.02535