Vane Bench Eval

vane-bench-eval

VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs — Gani et al. (2024) (arXiv:2406.10326, 2024)

What this evaluates

Evaluates the ability of video-language models to detect subtle, rapid, and contextually nuanced anomalies in both real-world surveillance footage and high-fidelity AI-generated videos. It probes fine-grained temporal reasoning, visual grounding, and robustness to synthetic artifacts through a multiple-choice question-answering format.

Datasets

• VANE-Bench — total 325; splits: test (325); repo https://github.com/rohit901/VANE-Bench

Metrics

• MC-Video QA accuracy (primary) — range: [0, 1]
  • Standard multiple-choice accuracy: the fraction of questions where the model's selected option exactly matches the ground-truth answer. Calculated as correct predictions divided by total questions.

Input / output format

Input: A video clip accompanied by a natural language question and a set of multiple-choice options.

Output: The model must select exactly one option from the provided multiple-choice list.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, golds):
    if pred == gold:
        correct += 1
accuracy = correct / len(golds)

Common pitfalls

• Anomalies are often subtle and rapid, requiring fine-grained temporal reasoning rather than relying on static frame-level inspection.
• AI-generated videos (e.g., SORA) have high visual fidelity, making anomalies extremely difficult to detect and potentially causing models to hallucinate or guess.
• Multiple-choice options may contain plausible distractors, so high scores can sometimes reflect guessing or textual priors rather than true visual understanding.

Evidence (verbatim from paper)

We adapted the VAD problem into a Multiple-Choice Video Question Answering (MC-Video QA) task to facilitate the evaluation of LMMs, allowing for a more granular assessment of their video content understanding.

Citation

@misc{gani2024vanebench,
  title={VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs},
  author={Gani et al. (2024)},
  year={2024},
  note={arXiv:2406.10326}
}

• arXiv: 2406.10326