Online Video Understanding Eval

online-video-understanding-eval

Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions — Zhang et al. (2026) (arXiv:2604.18459, 2026)

What this evaluates

Evaluates a model's ability to perform real-time, streaming video understanding and long-horizon reasoning. It probes low-latency perception, temporal alignment, evidence-aligned response timing, and causal reasoning across short to hour-long video sequences.

Datasets

• StreamingBench — total ?; splits: test (-1)
• OVOBench — total ?; splits: test (-1)
• RTVBench — total ?; splits: test (-1)
• OVBench — total ?; splits: test (-1)
• VideoMME — total ?; splits: test (-1)
• MLVU — total ?; splits: test (-1)
• LongVideoBench — total ?; splits: test (-1)
• LVBench — total ?; splits: test (-1)

Metrics

• QA accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Computed as (number of correct predictions / total number of questions) * 100. Follows official per-suite scoring protocols, which typically use exact match or LLM-based grading depending on the benchmark.

Input / output format

Input: Video frames sampled at a specified frame rate (e.g., 1 fps) or progressive clip sequences, combined with a natural language question/query. Input resolution is capped at 448×448.

Output: Textual answer to the question. For online benchmarks, responses may include timestamped evidence or deferred answers until sufficient future frames are observed.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Follows official benchmark scoring (exact match or LLM-as-judge)
        if is_correct(pred, gold): 
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Models must adhere to strict latency constraints and 'answer-when-ready' timing rules in online benchmarks, which many offline models fail to satisfy.
• Token budgets are tightly constrained (e.g., 32 frames, hierarchical aggregation tokens), so naive frame sampling or full-context processing will exceed limits and break the progressive evaluation protocol.

Evidence (verbatim from paper)

Our evaluation spans complementary online and offline long-video QA suites that jointly stress real-time perception, temporal alignment, and long-horizon reasoning. StreamingBench targets low-latency, timestamped queries under streaming constraints. OVOBench enforces answer-when-ready timing—models defer responses until sufficient future evidence (real-time perception, forward tracking, active responding). RTVBench and OVBench probe continuous perception and online spatio-temporal reasoning via multi-timestamp, hierarchical questions and Past/Current/Future anchoring. For offline long-form understanding, VideoMME, MLVU, LongVideoBench, and LVBench cover short clips to hour-long videos, emphasizing granular recall and cross-scale reasoning. We follow official scoring protocols (per-suite QA accuracy and aggregates); full task/metric definitions are in Appendix §A.5.

Citation

@misc{zhang2026progressive,
  title={Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.18459}
}

• arXiv: 2604.18459