Meetingbank Eval

meetingbank-eval

MeetingBank: A Benchmark Dataset for Meeting Summarization — Hu et al. (2023) (arXiv:2305.17529, 2023)

What this evaluates

Evaluates the quality of abstractive and extractive summarization systems on city council meeting transcripts and videos. It probes a model's ability to capture key decisions, maintain factual accuracy, and produce fluent, coherent, and non-redundant summaries.

Datasets

• MeetingBank — total 1366; splits: train (-1), test (200)

Metrics

• Average Score (primary) — range: [1, 5]
  • Mean of five 5-point Likert scale scores: informativeness, factuality, fluency, coherence, and redundancy. Higher values indicate better summary quality.

Input / output format

Input: Video segment (typically ≤30 minutes), corresponding transcript, and a system-generated summary.

Output: Five scores (1–5) for each criterion, plus an averaged total score.

Scoring recipe

def compute_average_score(predictions):
    criteria = ['informativeness', 'factuality', 'fluency', 'coherence', 'redundancy']
    scores = [predictions[c] for c in criteria]
    return sum(scores) / len(scores)

Common pitfalls

• Summaries must be presented in random order to evaluators to prevent order-based bias.
• Human evaluation is only performed on a subset of 200 randomly selected instances, not the full dataset.
• Evaluators assess both the video and transcript alongside the summary, which differs from text-only benchmarks.

Evidence (verbatim from paper)

The workers are asked to watch a video segment, typically 30 minutes or less, read the transcript, and then evaluate the quality of each system summary based on five criteria: informativeness, factuality, fluency, coherence, and redundancy. A 5-point Likert scale is used to evaluate each criterion. The scores are then averaged, and standard deviation is also reported. ... AVERAGE SCORE

Citation

@misc{hu2023meetingbank,
  title={MeetingBank: A Benchmark Dataset for Meeting Summarization},
  author={Hu et al. (2023)},
  year={2023},
  note={arXiv:2305.17529}
}

• arXiv: 2305.17529