qhjqhj00 video-to-c-eval
Video-ToC: Video Tree-of-Cue Reasoning — Tan et al. (2026) (arXiv:2604.20473, 2026)
What this evaluates
Evaluates fine-grained video understanding, spatio-temporal reasoning, and hallucination mitigation in multimodal large language models by testing their ability to locate key visual cues and answer questions across diverse video benchmarks.
Datasets
- VSI-Bench — total ?; splits: test (-1)
- VideoMMMU — total ?; splits: test (-1)
- MMVU — total ?; splits: test (-1)
- MVBench — total ?; splits: test (-1)
- TempCompass — total ?; splits: test (-1)
- VideoMME — total ?; splits: test (-1)
- VideoHallucer — total ?; splits: test (-1)
Metrics
Accuracy(primary) — range: percent- Percentage of correctly answered questions after extracting the model's prediction from the tags and comparing it to the ground truth label. Average accuracy is computed across all benchmarks.
Input / output format
Input: Video frames (uniformly sampled 16, 32, or 64 frames at 128x28x28 or 256x28x28 resolution) paired with a text question. A unified prompt is used, with task instructions specifying the expected output format based on question type (multiple-choice or numerical/regression).
Output: Text response containing the model's reasoning and final answer enclosed in tags. For multiple-choice questions, only the option letter is provided inside the tags; for numerical questions, the value is provided.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = 0
for pred, gold in zip(predictions, gold_labels):
ans = extract_between_tags(pred, '<answer>', '</answer>')
if ans is not None and normalize(ans) == normalize(gold):
correct += 1
return (correct / len(gold_labels)) * 100
Common pitfalls
- VideoMME must be evaluated without subtitle assistance to match the reported protocol.
- MMVU evaluation strictly uses the multiple-choice question subset, not the full dataset.
- Parsing the tags is required; models that output answers outside these tags will be scored as incorrect or fail to parse.
Evidence (verbatim from paper)
TABLE I: Accuracy comparison on three video reasoning benchmarks and three video general benchmarks. “Avg.” denotes average accuracy of the six benchmarks.
Citation
@misc{tan2026videotoc,
title={Video-ToC: Video Tree-of-Cue Reasoning},
author={Tan et al. (2026)},
year={2026},
note={arXiv:2604.20473}
}
- arXiv: 2604.20473