Svbench Eval

svbench-eval

SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding — Yang et al. (2025) (arXiv:2502.10810, 2025)

What this evaluates

Evaluates large vision-language models' ability to perform sustained temporal reasoning and context tracking across long-form streaming videos. It probes multi-turn dialogue continuity, temporal dependency handling, and complex reasoning skills like counterfactual analysis and spatio-temporal speculation.

Datasets

• SVBench — total 49979; splits: test (-1)

Metrics

• Overall Score (OS) (primary) — range: percent
  • Aggregates scores from Semantic Accuracy (SA), Contextual Coherence (CC), Logical Consistency (LC), Temporal Understanding (TU), and Informational Completeness (IC) via an LLM-based judge. The authors do not provide a symbolic formula; OS is computed by averaging or weighting the five criterion scores.
• METEOR — range: percent
  • Evaluates precision, recall, and alignment of words and phrases between the references and the ground truth by considering synonymy and stemming.
• GPT4-Score — range: percent
  • Assesses the accuracy of generated answers solely based on the semantic similarity between a single answer and the ground truth, evaluated by GPT-4.

Input / output format

Input: Video clip/frames, current question, and optionally a history of preceding QA pairs up to the current timestamp.

Output: Natural language answer to the question.

Scoring recipe

def compute_metrics(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        sa = llm_judge(pred, gold, 'Semantic Accuracy')
        cc = llm_judge(pred, gold, 'Contextual Coherence')
        lc = llm_judge(pred, gold, 'Logical Consistency')
        tu = llm_judge(pred, gold, 'Temporal Understanding')
        ic = llm_judge(pred, gold, 'Informational Completeness')
        os = aggregate(sa, cc, lc, tu, ic)
        meteor = compute_meteor(pred, gold)
        gpt4 = compute_gpt4_score(pred, gold)
        scores.append({'OS': os, 'METEOR': meteor, 'GPT4-Score': gpt4})
    return mean_over_scores(scores)

Common pitfalls

• Confusing single-instance QA evaluation with the paper's multi-turn dialogue setup, which requires feeding historical QA pairs as context.
• Overlooking the 80% probabilistic temporal jump rule in streaming evaluation, which tests handling of non-linear context transitions.
• Assuming standard static video QA metrics apply; streaming evaluation demands continuous context integration across evolving video segments.

Evidence (verbatim from paper)

Overall Score (OS) is derived by aggregating the scores from each aforementioned criterion. Semantic Accuracy (SA) evaluates the accuracy of the generated answers based on a holistic understanding. It considers not only the direct overlap with ground-truth answers but also the context, coherence, and overall relevance of the response to the question posed.

Citation

@misc{yang2025svbench,
  title={SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2502.10810}
}

• arXiv: 2502.10810