# Vane Bench Eval

> Evaluates the ability of video-language models to detect subtle, rapid, and contextually nuanced anomalies in both real-world surveillance footage and high-fidelity AI-generated videos. It probes fine-grained temporal reasoning, visual grounding, and robustness to synthetic artifacts through a multiple-choice question-answering format. Use when the user wants to benchmark on VANE-Bench, or asks about evaluating this task. Reports MC-Video QA accuracy.

- Skill: `qhjqhj00/vane-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/vane-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vane-bench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/vane-bench-eval

---


# vane-bench-eval

> VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs — Gani et al. (2024) (arXiv:2406.10326, 2024)

## What this evaluates

Evaluates the ability of video-language models to detect subtle, rapid, and contextually nuanced anomalies in both real-world surveillance footage and high-fidelity AI-generated videos. It probes fine-grained temporal reasoning, visual grounding, and robustness to synthetic artifacts through a multiple-choice question-answering format.

## Datasets

- **VANE-Bench** — total 325; splits: test (325); repo https://github.com/rohit901/VANE-Bench

## Metrics

- `MC-Video QA accuracy` **(primary)** — range: [0, 1]
  - Standard multiple-choice accuracy: the fraction of questions where the model's selected option exactly matches the ground-truth answer. Calculated as correct predictions divided by total questions.

## Input / output format

**Input**: A video clip accompanied by a natural language question and a set of multiple-choice options.

**Output**: The model must select exactly one option from the provided multiple-choice list.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, golds):
    if pred == gold:
        correct += 1
accuracy = correct / len(golds)
```

## Common pitfalls

- Anomalies are often subtle and rapid, requiring fine-grained temporal reasoning rather than relying on static frame-level inspection.
- AI-generated videos (e.g., SORA) have high visual fidelity, making anomalies extremely difficult to detect and potentially causing models to hallucinate or guess.
- Multiple-choice options may contain plausible distractors, so high scores can sometimes reflect guessing or textual priors rather than true visual understanding.

## Evidence (verbatim from paper)

> We adapted the VAD problem into a Multiple-Choice Video Question Answering (MC-Video QA) task to facilitate the evaluation of LMMs, allowing for a more granular assessment of their video content understanding.

## Citation

```bibtex
@misc{gani2024vanebench,
  title={VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs},
  author={Gani et al. (2024)},
  year={2024},
  note={arXiv:2406.10326}
}
```

- arXiv: 2406.10326