Agrigpt Omni Eval

agrigpt-omni-eval

AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence — Bo Yang et al. (2025) (arXiv:2512.10624, 2025)

What this evaluates

Evaluates a unified speech-vision-text model's capability in multilingual agricultural reasoning, covering text generation, vision-language QA, and multimodal speech understanding across open-ended and multiple-choice formats.

Datasets

• AgriBench-13K — total 13000; splits: test (-1)
• AgriBench-VL-4K — total 4000; splits: test (-1)
• AgriBench-Omni-2K — total 2000; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Fraction of correctly predicted answers on multiple-choice tasks. Calculated as the number of matches between model predictions and gold labels divided by the total number of instances.
• Pairwise Win Rate — range: percent
  • Proportion of head-to-head comparisons where the model's open-ended response is judged superior to a baseline's response on the same instance. Ties are counted separately and included in the denominator.

Input / output format

Input: Multimodal inputs combining speech audio, text prompts, and/or images in four configurations: speech-only, speech+text, speech+image, and speech+image+text.

Output: Text responses, either open-ended answers or selected multiple-choice options.

Scoring recipe

# Accuracy (Multiple Choice)
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
accuracy = correct / len(predictions)

# Pairwise Win Rate (Open QA)
wins = sum(1 for p, b in zip(preds, baselines) if judge(p) > judge(b))
ties = sum(1 for p, b in zip(preds, baselines) if judge(p) == judge(b))
losses = sum(1 for p, b in zip(preds, baselines) if judge(p) < judge(b))
win_rate = wins / (wins + ties + losses)

Common pitfalls

• Pairwise win rates are computed against specific baselines rather than as absolute scores, making cross-model comparisons dependent on the chosen reference models.
• The evaluation aggregates results across six languages but reports per-language and overall metrics, which can mask performance disparities in low-resource dialects like Sichuanese or Cantonese.
• Real-world robustness is measured via synthetic vs. human speech pairwise comparisons, which does not directly translate to standard accuracy metrics and may obscure absolute performance degradation.

Evidence (verbatim from paper)

We adopt Win Rate for open-ended generation tasks and Accuracy for multiple-choice tasks to provide a comprehensive view of model behavior under different reasoning constraints.

Citation

@misc{yang2025agrigptomni,
  title={AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence},
  author={Bo Yang et al. (2025)},
  year={2025},
  note={arXiv:2512.10624}
}

• arXiv: 2512.10624