qhjqhj00 closp-crisislandmark-eval
CLOSP: A Unified Semantic Space for SAR, MSI, and Text in Remote Sensing — Daniele Rege Cambrin et al. (2025) (arXiv:2507.10403, 2025)
What this evaluates
Evaluates cross-modal retrieval and zero-shot classification capabilities for remote sensing imagery (SAR and multispectral optical) paired with text descriptions. It probes how well unified semantic embeddings align heterogeneous geospatial data with natural language for crisis event and land cover analysis.
Datasets
- CrisisLandMark — total 647000; splits: train (-1), test (-1); repo https://github.com/DarthReca/closp
Metrics
nDCG@1000(primary) — range: percent- Normalized Discounted Cumulative Gain at cutoff K. Computed as the ratio of the DCG score to the Ideal DCG (IDCG). Relevance is binary based on IoU ≥ 0.5.
Precision@1000— range: percent- Fraction of retrieved images that are relevant (IoU ≥ 0.5) among the top 1000 results.
Recall@1000— range: percent- Fraction of all relevant images in the corpus that appear in the top 1000 retrieved results.
Macro-F1— range: percent- Macro-averaged F1-score across 12 land cover/crisis classes, computed as the unweighted mean of per-class F1-scores.
Input / output format
Input: Text query (crisis event or land cover description) and image(s) (Sentinel-1 SAR or Sentinel-2 multispectral/RGB). For retrieval: single text query vs. image corpus. For classification: single image with 12 class keywords.
Output: For retrieval: ranked list of image IDs/embeddings. For classification: binary presence/absence prediction for each of the 12 classes based on a cosine similarity threshold.
Scoring recipe
def score_retrieval(query_emb, image_embs, gold_boxes, K=1000):
scores = cosine_similarity(query_emb, image_embs)
ranked = argsort(scores, descending=True)[:K]
rel = [1 if iou(pred, gold) >= 0.5 else 0 for pred, gold in zip(ranked, gold_boxes)]
dcg = sum(rel[i] / log2(i + 2) for i in range(K))
idcg = sum(1 / log2(i + 2) for i in range(min(K, sum(rel))))
return (dcg / idcg) * 100
def score_classification(img_emb, class_embs, gold_labels):
sims = cosine_similarity(img_emb, class_embs)
threshold = mean(sims)
preds = [1 if s > threshold else 0 for s in sims]
return macro_f1(gold_labels, preds) * 100
Common pitfalls
- Using a fixed global threshold for zero-shot classification instead of the paper's model-specific mean-score thresholding strategy.
- Ignoring the IoU ≥ 0.5 relevance threshold for retrieval, which directly impacts Precision and Recall calculations.
- Comparing fine-tuned and non-fine-tuned baselines without noting their divergent behaviors, as some baselines degrade after fine-tuning.
Evidence (verbatim from paper)
The top-performing model, GeoCLOSP, achieves an nDCG@1000 of 57.76%, representing a nearly 20-point absolute improvement over the best baseline, SkyCLIP-T (37.88%).
Citation
@misc{cambrin2025closp,
title={CLOSP: A Unified Semantic Space for SAR, MSI, and Text in Remote Sensing},
author={Daniele Rege Cambrin et al. (2025)},
year={2025},
note={arXiv:2507.10403}
}
- arXiv: 2507.10403