Stream Omni Eval

stream-omni-eval

Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model — Zhang et al. (2025) (arXiv:2506.13642, 2025)

What this evaluates

Evaluates multimodal capabilities across visual understanding, speech interaction, and vision-grounded speech tasks. It measures accuracy on standard VQA and knowledge-grounded QA benchmarks, and uses LLM-based scoring for open-ended spoken interactions.

Datasets

• VQA-v2 — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• ScienceQA-IMG — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• SEED-Bench — total ?; splits: test (-1)
• LLaVA-Bench-in-the-Wild — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)
• Llama Questions — total ?; splits: test (-1)
• Web Questions — total ?; splits: test (-1)
• SpokenVisIT — total ?; splits: test (-1); HF ICTNLP/SpokenVisIT

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match or normalized accuracy measuring whether the model's generated response matches the ground-truth answer.
• GPT-4o score — range: [1, 5]
  • LLM-as-a-judge score from 1 to 5 assigned by GPT-4o to evaluate response quality on open-ended vision-grounded speech tasks.

Input / output format

Input: Image + text question (vision), audio + text prompt (speech), or image + audio question (vision-grounded speech).

Output: Text response for VQA/QA tasks; audio response for speech-to-speech settings.

Scoring recipe

if dataset in ['Llama Q.', 'Web Q.', 'VQA-v2', 'GQA', 'VizWiz', 'ScienceQA-IMG', 'TextVQA', 'POPE', 'MME', 'MMBench', 'SEED-Bench', 'LLaVA-Bench-in-the-Wild', 'MM-Vet']:
    score = 1.0 if normalize(prediction) == normalize(gold) else 0.0
    metric = mean(score)
elif dataset == 'SpokenVisIT':
    prompt = f'Score the response to the question on a scale of 1-5: Q: {question}, A: {prediction}'
    score = gpt4o_score(prompt)
    metric = mean(score)
return metric

Common pitfalls

• Speech-to-speech evaluations require transcribing generated audio with Whisper-large-v3 before scoring, which can introduce transcription errors.
• SpokenVisIT uses GPT-4o for scoring, which may vary based on prompt wording and model version.
• Vision benchmarks follow LLaVA's specific evaluation scripts; using standard VQA evaluation scripts may yield different results.

Evidence (verbatim from paper)

For speech evaluation, we assess the model’s knowledge-grounded speech interaction on spoken question answering benchmarks, Llama Questions (Llama Q.) [[62]] and Web Questions (Web Q.) [[63]], where the metric is the accuracy that whether the model’s response matches the ground-truth answer. Following Fang et al. [[9]], the evaluation for SpokenVisIT employs the GPT model (gpt-4o version) to assign a score ranging from 1 to 5 for response.

Citation

@misc{zhang2025streamomni,
  title={Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.13642}
}

• arXiv: 2506.13642