Cctvbench Eval

cctvbench-eval

CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs — Zhou et al. (2026) (arXiv:2604.20460, 2026)

What this evaluates

Probes multimodal LLMs' ability to answer binary questions about traffic videos while maintaining logical consistency across counterfactual video-question pairs. It specifically diagnoses failure modes like positive omission, negative hallucination, and mutual-exclusivity violations by enforcing a strict quadruple-level decision rule.

Datasets

• CCTVBench — total ?; splits: test (-1)

Metrics

• QuadAcc (primary) — range: percent
  • Percentage of quadruples where the model correctly answers Yes to (positive video, positive question), No to (positive video, negative question), No to (counterfactual video, positive question), and Yes to (counterfactual video, negative question). Enforces strict logical consistency across paired counterfactuals.
• BaAcc — range: percent
  • Standard binary accuracy on the primary positive video-question pairs.
• MCCScore — range: percent
  • Matthews Correlation Coefficient for binary classification, measuring the quality of binary predictions across all quadruple instances.

Input / output format

Input: A traffic video (or its counterfactual counterpart) paired with a binary question. The prompt is formatted as: 'You are given a traffic video and a question. Answer based only on the video with exactly one word: Yes or No. Question: {QUESTION}'.

Output: A single word: 'Yes' or 'No'.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions and gold are dicts with keys: v+q+, v+q-, v-q+, v-q-
    # Expected: v+q+=Yes, v+q-=No, v-q+=No, v-q-=Yes
    quad_correct = (
        predictions['v+q+'] == 'Yes' and
        predictions['v+q-'] == 'No' and
        predictions['v-q+'] == 'No' and
        predictions['v-q-'] == 'Yes'
    )
    ba_correct = (
        predictions['v+q+'] == gold['v+q+'] and
        predictions['v-q-'] == gold['v-q-']
    )
    return {'QuadAcc': quad_correct, 'BaAcc': ba_correct}

Common pitfalls

• High binary accuracy (BaAcc) or MCCScore does not guarantee logical consistency; models often pass isolated instances but fail the quadruple constraint.
• High rejection rates on counterfactuals (Reject@v-) or positive questions (Reject@q-) can mask positive omissions, where models incorrectly answer 'No' to true events.
• Confidence intervals are computed via scene-level bootstrap resampling (2,000 replicates), not instance-level, which may underestimate per-instance variance.

Evidence (verbatim from paper)

Tab. 1 shows that most models reach moderate binary QA quality, with BaAcc commonly in the 50–70% range and non-trivial MCCScore, yet QuadAcc remains low across the board.

Citation

@misc{zhou2026cctvbench,
  title={CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs},
  author={Zhou et al. (2026)},
  year={2026},
  note={arXiv:2604.20460}
}

• arXiv: 2604.20460