Video QA Eval

video-qa-eval

Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward — Ruohong Zhang et al. (2024) (arXiv:2404.01258, 2024)

What this evaluates

Evaluates video large multimodal models on question-answering tasks across multiple benchmark datasets. Probes the model's ability to understand video content, generate factually accurate long-form responses, and align with language model-derived preferences using direct preference optimization.

Datasets

• MSVD-QA — total ?; splits: test (-1)
• MSRVTT-QA — total ?; splits: test (-1)
• TGIF-QA — total ?; splits: test (-1)
• ActivityNet-QA — total ?; splits: test (-1)
• VIDAL-QA — total ?; splits: test (-1)
• WebVid-QA — total ?; splits: test (-1)
• SSV2-QA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Correctness is determined by ChatGPT evaluation against ground truth (existing benchmarks) or by a ChatGPT-assigned quality score ≥ 3 (proposed benchmarks).
• score — range: other
  • Continuous quality score (approximately 1-5 scale) assigned by ChatGPT based on response relevance, accuracy, clarity, and completeness when evaluated against GPT-4V generated captions as supporting evidence.

Input / output format

Input: Video clip (or sequence of frames) paired with a natural language question.

Output: Free-form text response answering the question.

Scoring recipe

def evaluate(model, video, question, ground_truth=None, use_proposed=False):
    if use_proposed:
        evidence = get_gpt4v_caption(video)
        prompt = f'Q: {question}\nEvidence: {evidence}\nPred: {model(video, question)}\nScore 1-5:'
        score = call_chatgpt(prompt, model='gpt-3.5-turbo-0301')
        acc = 1 if score >= 3 else 0
        return acc, score
    else:
        prompt = f'Q: {question}\nGT: {ground_truth}\nPred: {model(video, question)}\nCorrect?'
        is_correct = call_chatgpt(prompt, model='gpt-3.5-turbo-0613')
        return is_correct

Common pitfalls

• Different ChatGPT versions significantly impact absolute metric values, though overall model rankings remain relatively stable.
• Existing benchmarks use auto-generated questions that may be grammatically incorrect or nonsensical, and single-word ground truths that fail to evaluate long-form generation quality.
• Unfreezing the MLP projector during DPO training causes a notable performance drop compared to keeping it frozen.

Evidence (verbatim from paper)

We evaluate model performance on three benchmark datasets: MSVD-QA Chen & Dolan (2011), MSRVTT-QA Xu et al. (2016), and TGIF-QA Jang et al. (2017), using ChatGPT with version gpt-3.5-turbo-0611 to assess model predictions. ... Applying the our reward system in section 4, we report the score from ChatGPT, and a score value ≥ 3 will be considered correct for accuracy calculation.

Citation

@misc{zhang2024directpreferenceoptimization,
  title={Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward},
  author={Ruohong Zhang et al. (2024)},
  year={2024},
  note={arXiv:2404.01258}
}

• arXiv: 2404.01258