Video Oasis Eval

video-oasis-eval

Video-Oasis: Rethinking Evaluation of Video Understanding — Lim et al. (2026) (arXiv:2603.29616, 2026)

What this evaluates

This protocol audits video understanding benchmarks to measure genuine spatio-temporal reasoning versus shortcut reliance. It filters out samples solvable without video context and evaluates models under diagnostic conditions (e.g., blind, audio-only, center-frame) to quantify performance degradation and dependency on actual video content.

Datasets

• EgoSchema — total 500; splits: test (-1)
• ImplicitQA — total 766; splits: test (-1)
• VSI-Bench — total 2490; splits: test (-1)
• TVBench — total 2205; splits: test (-1)
• VCR-Bench — total 511; splits: test (-1)
• RTV-Bench — total 4608; splits: test (-1)
• Video-Holmes — total 1837; splits: test (-1)
• MINERVA — total 1358; splits: test (-1)
• MMR-V — total 1257; splits: test (-1)
• VideoMME — total 2700; splits: test (-1)
• MVBench — total 3000; splits: test (-1)
• LVBench — total 1345; splits: test (-1)
• LongVideoBench — total 1337; splits: test (-1)
• MLVU — total 502; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions on the filtered dataset or under specific diagnostic conditions. Calculated as (correct predictions / total evaluated samples) * 100.
• filtering_ratio — range: percent
  • Proportion of samples removed from the original benchmark because they are identified as shortcut-prone or solvable without video context. Calculated as (removed samples / original samples) * 100.
• performance_gap — range: percent
  • Absolute difference in accuracy between the original benchmark set and the Video-Oasis filtered set, indicating how much performance drops when shortcuts are removed.

Input / output format

Input: Video clips with associated multiple-choice or open-ended questions. For diagnostic tests, inputs are modified: visual input removed (Blind), audio transcript only (Audio), concatenated captions (Narrative), single center frame (Center-Frame), randomly shuffled frames (Frame Shuffling), or independent frame processing (Bag-of-Frames).

Output: Model predictions (selected option or generated answer) for each question.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_filtering_ratio(original_count, remaining_count):
    removed = original_count - remaining_count
    return (removed / original_count) * 100

def compute_performance_gap(acc_original, acc_remaining):
    return abs(acc_original - acc_remaining)

Common pitfalls

• Models may achieve high accuracy on original benchmarks by relying on textual priors or static frame cues rather than actual video understanding.
• Diagnostic tests like the Audio or Narrative tests can yield high scores if benchmarks contain redundant textual information, masking poor spatio-temporal reasoning.
• Filtering ratio does not strictly correlate with original accuracy, meaning high-performing benchmarks can still contain many shortcut samples.

Evidence (verbatim from paper)

MVBench achieves a higher original accuracy (71.2) than EgoSchema (62.4), yet EgoSchema exhibits both a higher filtering ratio (75.8 vs. 66.0) and a substantially larger performance gap (40.5 vs. 20.7).

Citation

@misc{lim2026videooasis,
  title={Video-Oasis: Rethinking Evaluation of Video Understanding},
  author={Lim et al. (2026)},
  year={2026},
  note={arXiv:2603.29616}
}

• arXiv: 2603.29616