Disasterm3 Eval

disasterm3-eval

DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response — Junjue Wang et al. (2025) (arXiv:2505.21089, 2025)

What this evaluates

Evaluates large vision-language models' ability to assess disaster damage from remote sensing imagery. It probes capabilities in multi-sensor (optical/SAR) understanding, object counting, relational reasoning, and generating professional disaster response reports.

Datasets

• DisasterM3 — total 26988; splits: Bench (-1), Instruct (-1)

Metrics

• accuracy (%) (primary) — range: percent
  • Percentage of correct answers on multiple-choice tasks including disaster scene recognition, type recognition, bearing body recognition, damaged building counting, damaged road estimation, and object relational reasoning.
• GPT-4.1 score — range: [1, 5]
  • Automated scoring by GPT-4.1 on a 5-point scale for open-ended tasks: damage assessment precision (DAP), damage detail recall (DDR), factual correctness (FC), recovery necessity (RN), strategic completeness (SC), and action priority precision (APP).
• mIoU — range: [0, 1]
  • Mean Intersection over Union across all referring segmentation categories (Road, Building, Other).
• cIoU — range: [0, 1]
  • Centered Intersection over Union for referring segmentation masks, following standard vision-language segmentation protocols.

Input / output format

Input: Paired bi-temporal remote sensing images (optical-optical or optical-SAR) accompanied by natural language instructions or questions.

Output: Multiple-choice selections, free-text disaster assessment reports or advice, or pixel-level segmentation masks for referred objects.

Scoring recipe

def score_qa(preds, golds):
    return sum(1 for p, g in zip(preds, golds) if p == g) / len(golds) * 100
def score_open(preds, golds, criteria):
    scores = [gpt41_eval(pred, gold, criteria) for pred, gold in zip(preds, golds)]
    return sum(scores) / len(scores)
def score_seg(pred_masks, gt_masks):
    ious = [compute_iou(p, g) for p, g in zip(pred_masks, gt_masks)]
    return sum(ious) / len(ious)

Common pitfalls

• Models exhibit significant domain gaps on disaster scenes due to lack of disaster-specific pretraining data.
• Performance drops substantially on SAR imagery compared to optical imagery due to reduced semantics and underrepresentation in generic VLMs.
• Damage counting accuracy is highly sensitive to object density, with fine-tuned models prone to overfitting in certain density ranges.
• Evaluation results vary noticeably with prompt phrasing, especially for smaller or less robust VLMs.

Evidence (verbatim from paper)

Following common settings*[[16], [34]], we adopted accuracy (%) for the multiple-choice tasks, i.e., disaster scene recognition (DSR), disaster type recognition (DTR), bearing body recognition (BBR), damaged building counting (DBC), damaged road estimation (DRE), object relational reasoning (ORR). The open-ended tasks are scored using GPT4.1 at a scale of 5 points. Disaster caption is measured from damage assessment precision (DAP), damage detail recall (DDR), and factual correctness (FC). Restoration advice is measured from recovery necessity (RN), strategic completeness (SC), and action priority precision (APP). The average accuracy (AVG) denotes the overall performance. Evaluation prompts are provided in Appendix § C. As for referring segmentation, we chose cIoU and mIoU following previous work[[15], [50]]*.

Citation

@misc{wang2025disasterm3,
  title={DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response},
  author={Junjue Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.21089}
}

• arXiv: 2505.21089