Mcif Eval

mcif-eval

MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks — Papi et al. (2025) (arXiv:2507.19634, 2025)

What this evaluates

Evaluates multimodal models' ability to follow crosslingual instructions on scientific talks, testing speech recognition, translation, question answering, and summarization across short and long contexts in English, German, Italian, and Chinese.

Datasets

• MCIF — total ?; splits: MCIFfix (-1), MCIFmix (-1)

Metrics

• WER — range: percent
  • Word Error Rate; measures the number of insertions, deletions, and substitutions required to change the predicted transcription into the reference, normalized by reference length.
• COMET — range: [-1, 1]
  • A neural machine translation evaluation metric that uses a pre-trained model to predict human judgments of translation quality based on source, hypothesis, and context.
• BERTScore (primary) — range: [0, 1]
  • Computes token-level precision, recall, and F1 score using contextual embeddings from BERT, comparing predicted text against the reference without strict lexical matching.

Input / output format

Input: Multimodal scientific talk inputs (speech audio, video frames, or text transcripts) paired with crosslingual instructions (recognition, translation, question answering, or summarization). Context length is specified as short or long.

Output: Text response generated by the model corresponding to the instruction (e.g., transcription, translated text, answer, or summary).

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'recognition':
        return compute_wer(predictions, golds)
    elif task == 'translation':
        return compute_comet(predictions, golds)
    elif task in ['question_answering', 'summarization']:
        return compute_bertscore(predictions, golds, normalize=False)
    return None

Common pitfalls

• Long-form context inputs cause significant performance degradation across most models, primarily due to undertranslation or context window limitations.
• Prompt variations (MCIFmix vs. MCIFfix) expose robustness issues; some models misinterpret instructions (e.g., performing image captioning instead of transcription on short-form inputs).
• SpeechLLMs generally struggle with long-form tasks compared to MLLMs, which maintain or improve performance.

Evidence (verbatim from paper)

We also report the non-normalized BERTScore results for question answering and summarization tasks in [Appendix E]. In recognition, SpeechLLMs show strong performance on short-form audio, confirming their specialization in speech transcription tasks. However, performance degrades significantly in the long-form scenario, where the WER increases substantially across all models.

Citation

@misc{papi2025mcif,
  title={MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks},
  author={Papi et al. (2025)},
  year={2025},
  note={arXiv:2507.19634}
}

• arXiv: 2507.19634