Omnimodal QA Eval

omnimodal-qa-eval

OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering — Zhu et al. (2026) (arXiv:2602.03707, 2026)

What this evaluates

This benchmark probes an agent's ability to perform long-horizon question answering over audio-video streams. It specifically tests fine-grained multimodal retrieval, multi-turn tool calling, and budget-aware reasoning when key evidence is scattered across time.

Datasets

• OmniVideoBench — total ?; splits: test (-1)
• WorldSense — total ?; splits: test (-1)
• Daily-Omni — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. The paper reports accuracy on each ability subset and averages them to measure overall performance across heterogeneous reasoning skills.

Input / output format

Input: Long audio-video streams paired with natural language questions.

Output: Textual answers to the questions.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Averaging accuracy across heterogeneous ability subsets may mask performance on specific reasoning skills.
• The exact answer verification method (string match vs. LLM judge) follows each benchmark's official protocol but is not explicitly detailed in the main text.
• Performance varies significantly with the retrieval budget (number of clips) and RL training state, so results are not static.

Evidence (verbatim from paper)

For these datasets, we report accuracy on each ability subset and their average score, which measures overall performance across heterogeneous reasoning skills.

Citation

@misc{zhu2026omniragagent,
  title={OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2602.03707}
}

• arXiv: 2602.03707