Causal2needles Eval

causal2needles-eval

Two Causally Related Needles in a Video Haystack — Li et al. (2025) (arXiv:2505.19853, 2025)

What this evaluates

Evaluates Video-Language Models' ability to perform joint retrieval and causal reasoning over two causally separated video clips connected by a 'bridge entity'. It also probes causal world modeling by asking models to identify cause-effect relationships in human behaviors within long videos.

Datasets

• Causal2Needles — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For 2-needle visual grounding questions, both Part 1 (effect clip) and Part 2 (cause clip) must be correctly identified to count as correct. Results are averaged over forward and reversed clip orders to mitigate positional bias.

Input / output format

Input: A vertically stacked image composed of 5 sampled frames per video clip, accompanied by the full narration text and a question. Clips are presented in a randomized span around the target cause and effect clips to prevent location shortcuts.

Output: A clip number (integer) indicating the location of the cause or effect event. Models may also output 'None' if they cannot locate the clip.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Positional bias: models heavily favor earlier clips in the sequence, artificially inflating accuracy for cause events in forward order.
• Static output bias: some open-source models output the same clip number regardless of the question content, leading to zero scores on complex queries.
• Textual bias: models may answer correctly using only the narration text without actually processing the video frames.
• Knowledge leakage: models might rely on memorized movie details from pretraining rather than visual grounding.

Evidence (verbatim from paper)

We report the accuracy on each type of questions. For VG 2-needle questions, we separately compute the accuracy for Part 1, Part 2, and both parts answered correctly. In addition, we evaluate each model using both the original (forward) and reversed video clip order, and report the average of the two orders as the final result on VG 2-needle questions.

Citation

@misc{li2025causal2needles,
  title={Two Causally Related Needles in a Video Haystack},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.19853}
}

• arXiv: 2505.19853