Geommbench Eval

geommbench-eval

GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing — Xiao et al. (2026) (arXiv:2604.08896, 2026)

What this evaluates

Evaluates expert-level multimodal intelligence in geoscience and remote sensing by testing domain knowledge, perceptual grounding, and spatiotemporal reasoning across diverse sensors, disciplines, and task complexities.

Datasets

• GeoMMBench — total 1053; splits: val (-1), test (-1)

Metrics

• Micro-averaged accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions, calculated by dividing the number of correct predictions by the total number of questions. Invalid outputs are marked incorrect.

Input / output format

Input: Image-based multiple-choice questions requiring identification, reasoning, or interpretation of geospatial and remote sensing data.

Output: Option letters (e.g., A, B, C, D) and/or numerical answers.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        extracted = extract_regex(pred)  # regex for option letters or numbers
        if extracted is None:
            continue  # invalid output marked incorrect
        if extracted == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Models may fail on visual tasks despite possessing strong textual geospatial knowledge, indicating visual-linguistic misalignment rather than true knowledge gaps.
• Domain-specialized MLLMs fine-tuned only on perception tasks often underperform general-purpose models on broader reasoning benchmarks.
• Text-only baselines can achieve high scores on knowledge questions but drop significantly when visual grounding is required, highlighting modality-specific evaluation needs.

Evidence (verbatim from paper)

Model responses are evaluated through an automated rule-based pipeline using regular expressions to extract option letters and numerical answers, with invalid outputs marked as incorrect. Micro-averaged accuracy serves as the primary metric.

Citation

@misc{xiao2026geommbench,
  title={GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing},
  author={Xiao et al. (2026)},
  year={2026},
  note={arXiv:2604.08896}
}

• arXiv: 2604.08896