Rs Vlm Zero Shot Eval

rs-vlm-zero-shot-eval

RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models — Junyao Ge et al. (arXiv:2408.14744, 2024)

What this evaluates

Evaluates the zero-shot image classification and text-to-image retrieval capabilities of Vision-Language Models (VLMs) fine-tuned on remote sensing data. It probes the model's ability to generalize to unseen RS scenes and text queries without task-specific fine-tuning, while also measuring resistance to catastrophic forgetting on general-domain benchmarks.

Datasets

• AID — total 2000; splits: test (2000)
• EuroSAT — total 2700; splits: test (2700)
• fMoW — total 106081; splits: val (106081)
• Million-AID — total 10000; splits: train (10000)
• PatternNet — total 30400; splits: train (30400)
• RESISC — total 31500; splits: train (31500)
• RSI-CB — total 24747; splits: train (24747)
• ImageNet-1K — total 50000; splits: val (50000)
• UCM Captions — total 210; splits: test (210)
• RSICD — total 1093; splits: test (1093)
• RSITMD — total 452; splits: test (452)

Metrics

• zero-shot top-1 accuracy (primary) — range: percent
  • Percentage of images correctly classified into their ground-truth category using the model's zero-shot classifier (typically via cosine similarity between image and text embeddings).
• zero-shot retrieval recall@K — range: percent
  • Recall at K (K=1, 5, 10), measuring the percentage of queries where the correct matching image or text appears within the top K results of the ranked list.

Input / output format

Input: Image patch (or full image) and corresponding text caption/query.

Output: Predicted class label (classification) or ranked list of images/texts (retrieval).

Scoring recipe

def compute_metrics(predictions, gold_labels, queries, gold_images, K=10):
    # Classification
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    accuracy = (correct / len(gold_labels)) * 100
    
    # Retrieval
    recalls = []
    for query, gold_img in zip(queries, gold_images):
        scores = model.encode(query) @ model.encode(images).T
        top_k_indices = np.argsort(scores)[-K:][::-1]
        recalls.append(1 if gold_img in top_k_indices else 0)
    recall_at_k = (sum(recalls) / len(queries)) * 100
    return accuracy, recall_at_k

Common pitfalls

• The paper uses non-standard splits for some benchmarks (e.g., fMoW on 'val', Million-AID/PatternNet/RESISC/RSI-CB on 'train') which differs from typical zero-shot evaluation protocols.
• Performance heavily depends on the initial CLIP checkpoint used for continual pre-training; results are not directly comparable across different base checkpoints without accounting for their distinct zero-shot baselines.
• Catastrophic forgetting is evaluated on ImageNet-1K separately from RS benchmarks, requiring careful tracking of general-domain performance degradation.

Evidence (verbatim from paper)

For evaluative purposes, we employ a series of zero-shot classification and image retrieval tasks. For zero-shot classification, we use eight widely recognized and comparatively extensive benchmark datasets, including AID, EuroSAT, fMoW, Million-AID, PatternNet, NWPU-RESISC45 (RESISC), RSI-CB256 (RSI-CB), and ImageNet-1K. ... The evaluation results are presented in Tables [9] and [10]. These tables report the top-1 accuracy for zero-shot classification tasks, as well as the average recall rates at top-1 (R@1), top-5 (R@5), and top-10 (R@10) for zero-shot text-to-image retrieval tasks.

Citation

@misc{ge2024rsteller,
  title={RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models},
  author={Junyao Ge et al.},
  year={2024},
  note={arXiv:2408.14744}
}

• arXiv: 2408.14744