Remoteshield Eval

remoteshield-eval

RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation — Rui Min et al. (arXiv:2604.17243, 2026)

What this evaluates

Probes the robustness and cross-condition consistency of multimodal large language models on Earth observation tasks under realistic visual and textual perturbations. Evaluates performance degradation and behavioral stability across clean and perturbed inputs for scene classification, VQA, and visual grounding.

Datasets

• RemoteShield clean-perturbed benchmarks — total ?; splits: test (-1); repo https://github.com/SteveJoker404/RemoteShield

Metrics

• RPD (primary) — range: other
  • Measures perturbation-induced degradation (lower is better). Exact mathematical formulation is provided in Sec. 3 of the paper.
• CCA (primary) — range: other
  • Measures cross-condition behavioral stability (higher is better). Exact mathematical formulation is provided in Sec. 3 of the paper.
• perturbed accuracy — range: percent
  • Standard accuracy computed on perturbed inputs for classification and VQA tasks.
• Acc@0.5 — range: percent
  • Visual grounding accuracy with Intersection over Union (IoU) threshold of 0.5.
• gIoU — range: other
  • Generalized Intersection over Union for visual grounding localization.

Input / output format

Input: Multimodal inputs consisting of Earth observation images paired with text queries/prompts. Inputs are provided in both clean and perturbed forms (e.g., cloud/fog visual degradation, colloquial textual noise, homoglyph substitution).

Output: Task-specific predictions: class labels for scene classification, natural language answers for VQA, and bounding box coordinates for visual grounding.

Scoring recipe

# Compute standard task metrics per instance
clean_acc = accuracy(clean_preds, gold)
perturbed_acc = accuracy(perturbed_preds, gold)

# Visual grounding specific
giou = compute_giou(pred_box, gt_box)
acc_at_05 = (giou >= 0.5).float().mean()

# RPD and CCA are aggregated across the benchmark per Sec. 3 protocol
# RPD: lower indicates less degradation from clean to perturbed
# CCA: higher indicates stronger cross-condition consistency

Common pitfalls

• RPD and CCA are the headline metrics, but standard task metrics (accuracy, Acc@0.5, gIoU) are also reported and sometimes used to claim performance gains.
• The evaluation covers both seen perturbations (cloud/fog, colloquial noise) and unseen perturbations (homoglyphs), which must be distinguished when comparing robustness.
• Ablation studies show that Mix-SFT can achieve a low RPD but fails on perturbed accuracy and CCA, indicating that low RPD alone does not guarantee robustness without consistency checks.

Evidence (verbatim from paper)

Because RemoteShield is designed to improve robustness and cross-condition consistency under multimodal perturbations, we center the analysis on RPD and CCA, using standard task metrics as supporting evidence. Across all three tasks, RemoteShield consistently achieves the lowest RPD and the highest CCA, indicating the smallest perturbation-induced degradation and the strongest behavioral stability among all compared models.

Citation

@misc{min2026remoteshield,
  title={RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation},
  author={Rui Min et al.},
  year={2026},
  note={arXiv:2604.17243}
}

• arXiv: 2604.17243