Moviesum Eval

moviesum-eval

MovieSum: An Abstractive Summarization Dataset for Movie Screenplays — Saxena et al. (2024) (arXiv:2408.06281, 2024)

What this evaluates

Evaluates the ability of abstractive summarization models to generate concise, coherent summaries of long, dispersed movie screenplay narratives. It probes long-document understanding, narrative coherence, and the model's capacity to synthesize information across thousands of tokens.

Datasets

• MovieSum — total ?; splits: test (-1); repo https://github.com/saxenarohit/MovieSum

Metrics

• ROUGE F1 (1/2/L) (primary) — range: [0, 1]
  • F1 score computed over unigram (ROUGE-1), bigram (ROUGE-2), and longest common subsequence (ROUGE-L) overlaps between the generated summary and the reference Wikipedia plot summary.
• BERTScore — range: [0, 1]
  • F1 score computed using contextual embeddings from BERT to measure semantic similarity between generated and reference texts, independent of exact lexical overlap.

Input / output format

Input: Full movie screenplay text (long document, varying token lengths depending on model context window)

Output: Generated abstractive summary text

Scoring recipe

def compute_metrics(predictions, references):
    # Compute ROUGE F1 for unigrams, bigrams, and LCS
    rouge_results = rouge_score(references, predictions, use_stemmer=True, use_aggregator=True)
    rouge_f1 = {k: v['fmeasure'] for k, v in rouge_results.items()}
    
    # Compute BERTScore F1
    p, r, f1 = bert_score.score(predictions, references, lang='en')
    bert_f1 = f1.mean().item()
    
    return {
        'rouge_1_f1': rouge_f1['rouge1'],
        'rouge_2_f1': rouge_f1['rouge2'],
        'rouge_l_f1': rouge_f1['rougeL'],
        'bertscore_f1': bert_f1
    }

Common pitfalls

• ROUGE F1 inherently favors longer summaries, so extractive baselines like Lead-1024 often outperform abstractive models simply due to length bias.
• Zero-shot models with large context windows (e.g., 16K) do not automatically attend to the full input, requiring chunking strategies or fine-tuning to achieve competitive performance.
• Evaluating only on the test set without a validation split may lead to overfitting when fine-tuning long-context models like LongT5 or LED.

Evidence (verbatim from paper)

Table[3] shows the summarization evaluation results using ROUGE F1 (1/2/L) scores (Lin, [2004]) and BERTScore (Zhang et al., [2019]) on MovieSum.

Citation

@misc{saxena2024moviesum,
  title={MovieSum: An Abstractive Summarization Dataset for Movie Screenplays},
  author={Saxena et al. (2024)},
  year={2024},
  note={arXiv:2408.06281}
}

• arXiv: 2408.06281