Geoheight Bench Eval

geoheight-bench-eval

GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing — Xuran Hu et al. (arXiv:2603.25565, 2026)

What this evaluates

Evaluates height-aware multimodal reasoning in remote sensing, probing models on pixel-level elevation retrieval, object-level relative height ranking, scene-level terrain relief analysis, and fine-grained height-aware mask generation. It specifically tests the model's ability to integrate vertical spatial priors with optical imagery for accurate numerical estimation and spatial segmentation.

Datasets

• GeoHeight-Bench — total ?; splits: test (-1)
• GeoHeight-Bench+ — total ?; splits: test (-1)

Metrics

• Numerical QA Accuracy (primary) — range: percent
  • For template-based questions, a numerical error threshold of 20% is applied: predictions falling within 20% of the ground truth value are marked correct. Open-ended questions are evaluated by an LLM judge (Qwen2.5-7B) for semantic accuracy and logical quality.
• mIoU — range: [0, 1]
  • Mean Intersection over Union across all prediction classes. Calculated as the average of IoU scores for each class in the segmentation mask.
• cIoU — range: [0, 1]
  • Cumulative Intersection over Union across all pixels/classes, aggregating true positives, false positives, and false negatives globally before computing the ratio.

Input / output format

Input: Remote sensing optical imagery (RGB) paired with text prompts for QA or mask generation instructions. Height-aware models also receive height/DEM data or geometric priors as additional input channels.

Output: Textual responses for QA tasks; pixel-level binary or multi-class masks for segmentation tasks.

Scoring recipe

def score_qa(pred, gt):
    val = extract_number(pred)
    return 1.0 if abs(val - gt) / gt <= 0.20 else 0.0

def score_seg(pred_mask, gt_mask):
    inter = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    return inter / union if union > 0 else 0.0

# mIoU = mean(score_seg for each class)
# cIoU = global_inter / global_union

Common pitfalls

• Numerical QA does not use exact match; it applies a 20% relative error threshold for correctness.
• Open-ended responses are judged by an LLM (Qwen2.5-7B) rather than automated string matching or regex.
• Segmentation metrics distinguish between mIoU (per-class average) and cIoU (global cumulative), with scene-level tasks explicitly split by object type (buildings vs. trees).

Evidence (verbatim from paper)

For template-based QA, to ensure objective evaluation, we utilize Qwen2.5-1.5B for post-processing extraction and set a numerical error threshold of 20%; predictions falling within this range are deemed correct. For open-ended questions, we employ the Qwen2.5-7B as a judge model to assess semantic accuracy and logical quality. For the mask generation task, following existing semantic segmentation research, we adopt mean Intersection over Union (mIoU) and cumulative Intersection over Union (cIoU) as our core evaluation metrics.

Citation

@misc{hu2026geoheightbench,
  title={GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing},
  author={Xuran Hu et al.},
  year={2026},
  note={arXiv:2603.25565}
}

• arXiv: 2603.25565