Mcts Vcb Eval

mcts-vcb-eval

Evaluating Multimodal Large Language Models on Video Captioning via Monte Carlo Tree Search — Linhao Yu et al. (2025) (arXiv:2506.11155, 2025)

What this evaluates

Evaluates multimodal large language models on fine-grained video captioning by measuring how well generated descriptions capture verified, detailed key points from videos. It probes the model's ability to reason about and describe specific visual details like color, quantity, and position.

Datasets

• MCTS-VCB — total 1765; splits: test (1765); repo https://github.com/tjunlp-lab/MCTS-VCB

Metrics

• F1 (primary) — range: [0, 1]
  • F1 is computed from Precision and Recall of key-point entailment. Precision is the ratio of model-extracted key points entailed in the reference key points. Recall is the ratio of reference key points entailed in the model's description. F1 = 2 * P * R / (P + R).

Input / output format

Input: Video frames (sampled at varying counts, e.g., 8, 16, 32, or 64) and a system prompt guiding the model to generate a detailed caption.

Output: A natural language video caption/description ($D_{\text{model}}$). Key points ($KP_{\text{model}}$) are subsequently extracted from this description by a judge LLM.

Scoring recipe

# 1. Extract key points from model caption using judge LLM
KP_model = extract_keypoints(D_model, judge_llm="Qwen2-VL-72B")
# 2. Get reference key points from benchmark
KP_ref = get_reference_keypoints(video_id)
# 3. Compute Precision: ratio of KP_model entailed in KP_ref
P = count_entailed(KP_model, KP_ref) / len(KP_model)
# 4. Compute Recall: ratio of KP_ref entailed in D_model
R = count_entailed(KP_ref, D_model) / len(KP_ref)
# 5. Compute F1
F1 = (2 * P * R) / (P + R) if (P + R) > 0 else 0.0

Common pitfalls

• Using n-gram overlap metrics (ROUGE, BERTScore) instead of the key-point entailment method yields significantly lower correlation with human judgments.
• Model performance is highly sensitive to the number of input frames; exceeding the maximum frame count used during training can cause performance plateaus or drops.
• The evaluation relies on a specific judge LLM (Qwen2-VL-72B/2.5-72B-Instruct) and prompt to extract key points and judge entailment; deviating from this setup changes results.

Evidence (verbatim from paper)

First, for each evaluated model, we used the prompt shown in Figure [13] to guide Qwen2-VL-72B in extracting key points $KP_{\text{model}}$ from the model-generated description $D_{\text{model}}$. For each video, MCTS-VCB contains the verified video key points $KP_{\text{ref}}$. Then, we calculated the ratio of key points in $KP_{\text{model}}$ that are entailed in $KP_{\text{ref}}$ as Precision, and the ratio of key points in $KP_{\text{ref}}$ that are entailed in $D_{\text{model}}$ as Recall. Finally, we calculated F1 by $\frac{2PR}{P+R}$.

Citation

@misc{yu2025mctsvcbeval,
  title={Evaluating Multimodal Large Language Models on Video Captioning via Monte Carlo Tree Search},
  author={Linhao Yu et al. (2025)},
  year={2025},
  note={arXiv:2506.11155}
}

• arXiv: 2506.11155