Rsmeb Eval

rsmeb-eval

VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing — Emanuel Sánchez Aimar et al. (2025) (arXiv:2512.11490, 2025)

What this evaluates

Evaluates a vision-language model's ability to perform zero-shot classification, cross-modal retrieval, visual question answering, and fine-grained spatial grounding (including region-caption retrieval and geo-localization) on remote sensing imagery. It measures how well instruction-conditioned contrastive pretraining aligns multimodal features with geospatial metadata and textual prompts.

Datasets

• AID — total ?; splits: test (-1)
• Million-AID — total ?; splits: test (-1)
• RSI-CB — total ?; splits: test (-1)
• EuroSAT — total ?; splits: test (-1)
• UCM — total ?; splits: test (-1)
• PatternNet — total ?; splits: test (-1)
• RSITMD — total ?; splits: test (-1)
• RSICD — total ?; splits: test (-1)
• UCM-caption — total ?; splits: test (-1)
• LRBEN — total ?; splits: test (-1)
• HRBEN — total ?; splits: test (-1)

Metrics

• accuracy — range: percent
  • Percentage of correctly classified images out of the total test set.
• recall@1/5/10 (R@k) — range: percent
  • Fraction of queries where the ground-truth item appears in the top-k retrieved results. The paper reports the average of R@1, R@5, and R@10 for cross-modal retrieval.
• precision@1 (P@1) — range: percent
  • Fraction of queries where the top-1 retrieved result matches the ground truth. Used for region-based CIR, VQA, visual grounding, spatial localization, and semantic geo-localization.
• Friedman score (primary) — range: other
  • Average rank of a method across all datasets/tasks in a meta-task. Lower scores indicate better overall performance.

Input / output format

Input: Multimodal inputs including satellite images, text prompts/questions, bounding box coordinates, and geographic coordinates (latitude/longitude). Inputs are truncated to 4,096 tokens. For classification, a 20-prompt ensemble is used.

Output: Class labels for classification; ranked lists of retrieved images or texts for retrieval tasks; bounding boxes or image IDs for grounding/localization; and text answers for VQA.

Scoring recipe

def compute_metric(predictions, golds, metric_type, k=1):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif metric_type in ['recall', 'precision']:
        top_k = predictions[:k]
        return sum(1 for g in golds if g in top_k) / len(golds)
    elif metric_type == 'avg_recall':
        r1 = sum(1 for g in golds if g in predictions[:1]) / len(golds)
        r5 = sum(1 for g in golds if g in predictions[:5]) / len(golds)
        r10 = sum(1 for g in golds if g in predictions[:10]) / len(golds)
        return (r1 + r5 + r10) / 3
    elif metric_type == 'friedman':
        return sum(ranks) / len(ranks)

Common pitfalls

• Evaluating specialized baselines (e.g., RemoteCLIP) in-distribution while testing the proposed model zero-shot, which inflates baseline scores.
• Using a single prompt for classification instead of the recommended 20-prompt ensemble, which significantly underestimates embedding model accuracy.
• Applying instruction prompts to dual-encoder baselines, which the authors note typically hurts their performance since they were not trained with instructions.

Evidence (verbatim from paper)

We evaluate each task using metrics according to the literature: accuracy for classification; recall@1/5/10 (R@k) and the average of the three metrics for cross‑modal retrieval [[26], [45]]; and precision@1 (P@1) for region-based CIR, VQA, visual grounding, spatial localization, and semantic geo‑localization [[15]].

Citation

@misc{sanchezaimar2025vlm2geovec,
  title={VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing},
  author={Emanuel Sánchez Aimar et al. (2025)},
  year={2025},
  note={arXiv:2512.11490}
}

• arXiv: 2512.11490