Streamingbench Eval

streamingbench-eval

StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding — Lin et al. (2024) (arXiv:2411.03628, 2024)

What this evaluates

Evaluates multimodal large language models' ability to understand real-time streaming video, integrate visual and audio information, maintain contextual continuity across sequential questions, and proactively output information at specific timestamps.

Datasets

• StreamingBench — total 4500; splits: test (4500); repo https://github.com/THUNLP-MT/StreamingBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. For the Proactive Output (PO) task, accuracy is computed as the proportion of queries where the model's actual output timestamp falls within a 2-second margin of the ground truth timestamp.

Input / output format

Input: Video frames extracted from the beginning of the video up to the timestamp of the current question. For Sequential Question Answering (SQA), previous QA pairs are appended as text history: "{Timestamp1}: {QA1} …; Answer the question accordingly: {current question}".

Output: Text answer for multiple-choice questions. For Proactive Output, a timestamp and relevant keywords.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_po_accuracy(pred_timestamps, gold_timestamps):
    correct = sum(1 for p, g in zip(pred_timestamps, gold_timestamps) if abs(p - g) < 2.0)
    return (correct / len(gold_timestamps)) * 100

Common pitfalls

• Streaming tasks are evaluated offline by clipping the video to the question timestamp, which may not reflect true real-time processing capabilities.
• The Proactive Output task requires a specific polling strategy (querying every second) rather than a single forward pass, making direct comparison with standard benchmarks difficult.
• Contextual understanding tasks require manually appending QA history to the input, which is not natively supported by most MLLMs.

Evidence (verbatim from paper)

We use accuracy as the evaluation metric for all multiple-choice questions. For the Proactive Output task, a question is considered accurately resolved only if the difference between the actual output timestamp and the ground truth timestamp is less than two seconds. The average accuracy across all queries is then computed and used as the performance metric for the PO task.

Citation

@misc{lin2024streamingbench,
  title={StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2411.03628}
}

• arXiv: 2411.03628