Streambench Eval

streambench-eval

Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge — Haomiao Xiong et al. (2025) (arXiv:2501.13468, 2025)

What this evaluates

Evaluates real-time streaming video understanding and multi-turn dialogue capabilities. It measures semantic correctness, factual accuracy, dialogue coherence, and system latency across diverse video types and query formats.

Datasets

• STREAMBENCH — total ?; splits: test (-1); repo https://github.com/hmxiong/StreamChat

Metrics

• Acc. (primary) — range: [0, 1]
  • Accuracy percentage of the model's response matching the ground truth or expected answer.
• Sco. — range: [0, 5]
  • Semantic correctness score assigned by LLaMA-3 on a scale of [0, 5], where higher scores indicate closer alignment with expected answers.
• Coh. — range: other
  • Coherence score measured by the fluctuation of Sco. across dialogue turns; smaller fluctuations indicate smoother dialogue.
• RPD — range: seconds
  • Request Processing Delay in seconds, calculated as the time from user request submission to the start of response generation.
• FPS — range: other
  • Frames per second processed by the streaming system.

Input / output format

Input: Streaming video frames (processed in chunks/groups) concatenated with a user query or multi-turn dialogue history.

Output: Natural language text response to the user query.

Scoring recipe

def compute_metrics(predictions, golds, timestamps):
    # Sco.: LLaMA-3 judge scores response vs expected answer [0,5]
    sco = [llama3_score(p, g) for p, g in zip(predictions, golds)]
    # Acc.: Binary match or percentage
    acc = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    # Coh.: Standard deviation of Sco. across turns (lower is better)
    coh = [np.std(turn_scores) for turn_scores in predictions]
    # RPD: Time from request to start of generation
    rpd = [ts['start_gen'] - ts['request'] for ts in timestamps]
    return {'Sco.': np.mean(sco), 'Acc.': np.mean(acc), 'Coh.': np.mean(coh), 'RPD': np.mean(rpd)}

Common pitfalls

• Coherence (Coh.) is not a direct quality score but a measure of score fluctuation across turns; lower values indicate better coherence.
• RPD only measures latency until the start of response generation, not the full generation time.
• Semantic scoring (Sco.) relies on LLaMA-3 as a judge, which may introduce bias or differ from human evaluation.

Evidence (verbatim from paper)

We evaluate semantic similarity in single conversations using the LLaMA-3 model, which assigns a semantic correctness score (Sco.) ranging from [0,5], where higher scores reflect responses that more closely align with the expected answers. For assessing coherence in multi-turn dialogues, we compute score fluctuations across turns; smaller fluctuations (Coh.) indicate a smoother dialogue experience. Additionally, we measure request processing delay (RPD), defined as the time (in seconds) from user request submission to the start of response generation.

Citation

@misc{xiong2025streamchat,
  title={Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge},
  author={Haomiao Xiong et al. (2025)},
  year={2025},
  note={arXiv:2501.13468}
}

• arXiv: 2501.13468