Omnigaiatoolreasoning Eval

omnigaiatoolreasoning-eval

OmniGAIA: Towards Native Omni-Modal AI Agents — Xiaoxi Li et al. (2026) (arXiv:2602.22897, 2026)

What this evaluates

Evaluates native omni-modal AI agents' ability to perform multi-hop cross-modal reasoning across video, audio, and image inputs. It probes their capacity to integrate external tools (web search, browser, code execution) for evidence gathering and to produce verifiable open-form answers under varying task difficulties.

Datasets

• OmniGAIA — total ?; splits: test (-1); repo https://github.com/RUC-NLPIR/OmnigAIA

Metrics

• Pass@1 (primary) — range: percent
  • The percentage of tasks where the model's final answer is judged semantically equivalent to the ground truth. Equivalence is determined by an LLM-as-a-Judge (DeepSeek-V3.2) using a standardized prompt.

Input / output format

Input: Multi-modal inputs (video, audio, images) containing real-world events, accompanied by multi-hop natural language queries. Models are provided access to external tools (web search, browser, code executor).

Output: A final open-form textual answer to the query.

Scoring recipe

def compute_pass_at_1(predictions, gold_answers, judge_model="DeepSeek-V3.2"):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if judge_model.evaluate_equivalence(pred, gold):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• LLM-as-a-Judge equivalence judgments are prompt-sensitive; results depend heavily on the specific judging prompt design.
• High tool-call frequency does not guarantee success; models may exhibit 'thrashing' behaviors that waste budget without improving accuracy.
• Native perception vs. tool-based perception trade-offs are task-dependent; tools can help on Easy/Medium tasks but degrade Hard task performance.

Evidence (verbatim from paper)

Pass@1 is reported, where a trial is considered correct if the model's final answer is judged equivalent to the ground truth. The judging prompt is detailed in Appendix B. All models are provided with the same external tools, including web search, browser, and code executor.

Citation

@misc{li2026omnigaiatowards,
  title={OmniGAIA: Towards Native Omni-Modal AI Agents},
  author={Xiaoxi Li et al. (2026)},
  year={2026},
  note={arXiv:2602.22897}
}

• arXiv: 2602.22897