Mm Uavbench Eval

mm-uavbench-eval

MM-UAVBench: How Well Do Multimodal Large Language Models See, Think, and Plan in Low-Altitude UAV Scenarios? — Dai et al. (2025) (arXiv:2512.23219, 2025)

What this evaluates

Evaluates Multimodal Large Language Models on low-altitude UAV scenarios, probing their capabilities in visual perception, multi-view spatial reasoning, and egocentric/exocentric planning across diverse real-world aerial imagery tasks.

Datasets

• MM-UAVBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Computed as (number of correct predictions / total number of questions) × 100.

Input / output format

Input: Multiple-choice questions accompanied by UAV imagery (single or multi-view), requiring the model to identify objects, classify orientations, analyze intents, or plan agent movements.

Output: Selection of the correct option from a fixed set of multiple-choice answers.

Scoring recipe

correct_count = 0
for trial in range(3):
    preds = model.generate(inputs, top_p=1.0, temperature=0.0, num_beams=3)
    correct_count += sum(1 for p, g in zip(preds, gold) if p == g)
final_score = (correct_count / (3 * len(gold))) * 100

Common pitfalls

• Multi-view evaluation often yields lower accuracy than the best single-view input, indicating a '1+1<2' fusion failure rather than additive benefit.
• Egocentric planning (UAV self-motion) is consistently harder than exocentric planning (other agents), revealing embodiment grounding gaps.
• Object scale heavily biases results; small targets in the field of view cause significant accuracy drops across all models.

Evidence (verbatim from paper)

All questions in MM-UAVBENCH are designed in a multiple-choice format. We report accuracy as the primary evaluation metric. Each model is evaluated three times, and the average accuracy is taken as the final score for each task. For reproducibility, we use a greedy decoding configuration with top_p = 1.0, temperature = 0.0, and num_beams = 3.

Citation

@misc{dai2025mmuavbench,
  title={MM-UAVBench: How Well Do Multimodal Large Language Models See, Think, and Plan in Low-Altitude UAV Scenarios?},
  author={Dai et al. (2025)},
  year={2025},
  note={arXiv:2512.23219}
}

• arXiv: 2512.23219