Mmla Eval

mmla-eval

Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark — Zhang et al. (2025) (arXiv:2504.16427, 2025)

What this evaluates

Evaluates the ability of LLMs and MLLMs to perform multimodal language analysis across six high-level semantic dimensions: intent, emotion, dialogue act, sentiment, speaking style, and communication behavior. It probes cross-modal reasoning and cognitive-level semantic understanding in conversational contexts using aligned text and video utterances.

Datasets

• MIntRec — total ?; splits: train (-1), val (-1), test (-1)
• MIntRec2.0 — total ?; splits: train (-1), val (-1), test (-1)
• MELD — total ?; splits: train (-1), val (-1), test (-1)
• IEMOCAP — total ?; splits: train (-1), val (-1), test (-1)
• MOSI — total ?; splits: train (-1), val (-1), test (-1)
• CH-SIMS v2.0 — total ?; splits: train (-1), val (-1), test (-1)
• UR-FUNNY-v2 — total ?; splits: train (-1), val (-1), test (-1)
• MUStARD — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (ACC) (primary) — range: [0, 1]
  • The proportion of correctly predicted labels out of the total number of instances. Calculated as (True Positives + True Negatives) / Total Instances.
• weighted F1-score (WF1) — range: [0, 1]
  • The harmonic mean of precision and recall, weighted by the number of true instances for each class.
• weighted precision (WP) — range: [0, 1]
  • The ratio of true positive predictions to the total number of positive predictions, weighted by class support.
• macro F1-score (F1) — range: [0, 1]
  • The unweighted mean of F1-scores calculated for each class independently.
• recall (R) — range: [0, 1]
  • The ratio of true positive predictions to the total number of actual positives.
• precision (P) — range: [0, 1]
  • The ratio of true positive predictions to the total number of predicted positives.

Input / output format

Input: Aligned text and video data at the utterance level for speakers.

Output: Classification label corresponding to the target semantic dimension (e.g., intent, emotion, dialogue act, sentiment, speaking style, or communication behavior).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

Common pitfalls

• Models must process both text and video modalities; evaluating only on text ignores the multimodal alignment requirement.
• The benchmark covers six distinct semantic dimensions, so results should be reported per dimension rather than aggregated blindly.
• Zero-shot performance is notably low; fine-tuning (SFT/IT) is required to reach 60-70% accuracy, so comparing only zero-shot baselines is misleading.

Evidence (verbatim from paper)

We employ six commonly used metrics: accuracy (ACC), weighted F1-score (WF1), weighted precision (WP), macro F1-score (F1), recall (R), and precision (P) for evaluation, as suggested in the literature. In particular, we report the primary results of ACC in this paper, with additional results for the remaining metrics provided in the Appendices.

Citation

@misc{zhang2025mmla,
  title={Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2504.16427}
}

• arXiv: 2504.16427