Cogstream Eval

cogstream-eval

CogStream: Context-guided Streaming Video Question Answering — Zhao et al. (2025) (arXiv:2506.10516, 2025)

What this evaluates

Evaluates a model's ability to perform question answering on streaming video by selectively retrieving and leveraging relevant historical context while ignoring irrelevant or noisy dialogue. It probes temporal reasoning, context awareness, and robustness to interference in long-form video understanding.

Datasets

• CogStream — total ?; splits: train (-1), test (-1); repo https://github.com/LiamZhao326/CogStream

Metrics

• Average of IA, DC, CA, TP, LC (primary) — range: [0, 100]
  • Average of five LLM-based scores: Information Accuracy (IA), Detail Completeness (DC), Context Awareness (CA), Temporal Precision (TP), and Logical Consistency (LC). Each metric is scored on a 0–100 scale by an LLM judge, and the final score is the arithmetic mean of the five.

Input / output format

Input: A video segment (typically sampled at 20 fps or 1 fps depending on the model) interleaved with historical question-answer pairs, followed by a natural language question about the current video content.

Output: A natural language text answer generated by the model.

Scoring recipe

def score(predictions, golds, llm_judge):
    scores = []
    for pred, gold in zip(predictions, golds):
        ia = llm_judge.evaluate(pred, gold, criterion="IA")
        dc = llm_judge.evaluate(pred, gold, criterion="DC")
        ca = llm_judge.evaluate(pred, gold, criterion="CA")
        tp = llm_judge.evaluate(pred, gold, criterion="TP")
        lc = llm_judge.evaluate(pred, gold, criterion="LC")
        scores.append((ia + dc + ca + tp + lc) / 5.0)
    return sum(scores) / len(scores)

Common pitfalls

• Frame sampling rates are not standardized across all baselines (e.g., 20 frames/segment vs. 1fps vs. 50 max), which can artificially inflate or deflate performance.
• LLM-based scoring relies heavily on the prompt template and judge model version; minor prompt changes can shift scores significantly.
• The hierarchical task breakdown (Basic/Streaming/Global) requires strict adherence to context-retrieval protocols; models that ingest all history without filtering often degrade on Streaming/Global tasks due to noise.

Evidence (verbatim from paper)

Inspired by SVbench, we enhance the LLM-based VQA metric (GPT4-score) for evaluation. We introduce the following metrics: Information Accuracy (IA), Detail Completeness (DC), Context Awareness (CA), Temporal Precision (TP), and Logical Consistency (LC). Each is scored between 0 and 100, and we report their average. See our Appendix for details of these metrics.

Citation

@misc{zhao2025cogstream,
  title={CogStream: Context-guided Streaming Video Question Answering},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2506.10516}
}

• arXiv: 2506.10516