M2k Vdg Eval

m2k-vdg-eval

M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation — Hongcheng Liu et al. (2024) (arXiv:2402.11875, 2024)

What this evaluates

Evaluates a model's ability to generate fluent, semantically accurate, and hallucination-free natural language responses grounded in video and audio inputs. It probes multimodal fusion, knowledge grounding, and dialogue generation capabilities across diverse question types and modalities.

Datasets

• AVSD10 — total ?; splits: test (-1)
• NExT-OE — total ?; splits: test (-1)
• MUSIC-AVQA — total ?; splits: test (-1)

Metrics

• BLEU1-4 — range: [0, 1]
  • N-gram precision scores (1-gram to 4-gram) comparing generated text to reference responses. BLEU4 is the standard n-gram overlap metric for text generation.
• METEOR — range: [0, 1]
  • Metric for Evaluation of Translation with Explicit ORdering, weighing precision, recall, and fragmentation penalty based on synonym and stem matching.
• ROUGH-L — range: [0, 1]
  • Longest Common Subsequence (LCS) based recall/precision score. Note: The paper misspells this as ROUGH-L instead of ROUGE-L.
• CIDEr (primary) — range: [0, 1]
  • Consensus-based Image Description Evaluation. Computes TF-IDF weighted n-gram similarity between generated and reference captions, emphasizing consensus with human references.
• WUPS Score — range: [0, 1]
  • Weighted Unambiguous Precision and Recall Score. Measures semantic similarity based on word overlap aligned via a knowledge hierarchy (e.g., WordNet).
• Accuracy — range: [0, 1]
  • Exact match or fuzzy match rate for question-answering tasks, measuring the percentage of correctly generated answers against ground truth.

Input / output format

Input: Multimodal context consisting of video frames/clips and corresponding audio tracks, paired with a conversational history or specific question.

Output: A single natural language sentence or short paragraph representing the model's generated dialogue response.

Scoring recipe

def evaluate(predictions, references):
    bleu4 = nltk.bleu([refs], preds, weights=(0.25, 0.25, 0.25, 0.25))
    meteor = compute_meteor(references, predictions)
    rouge_l = compute_rouge_l(references, predictions)  # Paper calls it ROUGH-L
    cider = compute_cider(references, predictions)
    wups = compute_wups(references, predictions)
    acc = sum(1 for p, r in zip(predictions, references) if match(p, r)) / len(predictions)
    return {'BLEU4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l, 'CIDEr': cider, 'WUPS': wups, 'Accuracy': acc}

Common pitfalls

• The paper misspells ROUGE-L as 'ROUGH-L' throughout the text and tables.
• Metrics are reported across three distinct datasets (AVSD10, NExT-OE, MUSIC-AVQA) with different modalities and question types; results cannot be averaged or directly compared across datasets.
• Accuracy for MUSIC-AVQA does not specify whether exact match or fuzzy matching is used, which significantly impacts reported scores.
• WUPS Score requires a specific knowledge hierarchy (e.g., WordNet) for alignment; results vary heavily depending on the implementation version.

Evidence (verbatim from paper)

Furthermore, we adopt different evaluation metrics from various aspects, including fluent expression (BLEU1-4, METEOR, ROUGH-L, CIDEr), semantic representation (WUPS Score $^{1}$ ), and accurate generation (Accuracy).

Citation

@misc{liu2024m2kvdg,
  title={M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation},
  author={Hongcheng Liu et al. (2024)},
  year={2024},
  note={arXiv:2402.11875}
}

• arXiv: 2402.11875