M3av Eval

m3av-eval

M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset — Chen et al. (2024) (arXiv:2403.14168, 2024)

What this evaluates

Evaluates multimodal academic lecture understanding across speech recognition, speech synthesis, and slide/script generation. It probes models' ability to handle complex academic language, rare words, multimodal alignment, and knowledge comprehension.

Datasets

• M3AV — total 367; splits: dev (-1), test (-1)

Metrics

• CER — range: percent
  • Character Error Rate: normalized edit distance between predicted and reference character sequences.
• WER — range: percent
  • Word Error Rate: normalized edit distance between predicted and reference word sequences.
• BWER (primary) — range: percent
  • Biasing Word Error Rate: total rare word errors divided by the total number of rare words, including insertions of rare words.
• FFE — range: other
  • F0 Frame Error: measures pitch contour deviation between synthesized and reference speech.
• MCD — range: other
  • Mel-Cepstral Distortion: measures spectral distortion between synthesized and reference speech.
• MOS — range: [1, 5]
  • Mean Opinion Score: subjective quality rating on a scale of 1 to 5 by human listeners.
• ROUGE-1/2/L (primary) — range: percent
  • Recall-oriented understudy for gisting evaluation: measures n-gram (1, 2) and longest common subsequence overlap between generated and reference text.

Input / output format

Input: Audio recordings of academic lectures, optionally paired with slide images/OCR text and reference paper sentences for knowledge supplementation.

Output: Transcriptions (ASR/CASR), synthesized audio waveforms (TTS), or generated text for slides/scripts (SSG).

Scoring recipe

def compute_cer_wer_bwer(pred, ref, rare_words):
    cer = edit_distance(pred, ref) / len(ref)
    wer = edit_distance(pred.split(), ref.split()) / len(ref.split())
    rare_errors = sum(1 for w in pred.split() if w not in ref.split() and w in rare_words)
    bwer = rare_errors / len(rare_words)
    return cer, wer, bwer

def compute_rouge(pred, ref):
    return rouge_score(ref, pred, use_stemmer=True)

Common pitfalls

• Whisper models are excluded from direct comparison due to data contamination and outputting written forms only.
• BWER specifically counts rare word errors, not overall WER, and includes insertions of rare words.
• MOS is subjective and requires human evaluation on a 1-5 scale, not an automated metric.

Evidence (verbatim from paper)

In addition to the character error rate (CER) and word error rate (WER) for ASR, we use biasing WER (BWER) for CASR. BWER is defined as the total rare word errors divided by the total number of rare words, including insertions of rare words.

Citation

@misc{chen2024m3av,
  title={M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.14168}
}

• arXiv: 2403.14168