Croma Eval

croma-eval

CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders — Fuller et al. (2023) (arXiv:2311.00566, 2023)

What this evaluates

Evaluates self-supervised remote sensing representations across classification and segmentation tasks using optical and radar-optical inputs. Probes representation quality via finetuning, linear/nonlinear probing, kNN, and clustering.

Datasets

• BigEarthNet — total 153485; splits: train (35420), val (118065)
• fMoW-Sentinel — total 156226; splits: train (71287), val (84939)
• EuroSAT — total 21600; splits: train (16200), val (5400)
• Canadian Cropland — total 76972; splits: train (53884), val (23088)
• DFC2020 — total 55026; splits: train (46152), val (8874)
• DW-Expert — total 71444; splits: train (20422), val (51022)
• MARIDA — total 3297; splits: train (1682), val (1615)

Metrics

• mAP (primary) — range: [0, 1]
  • Mean Average Precision across all classes. Computes the average precision for each class and averages them, suitable for multi-label classification.
• Top 1 Acc. (primary) — range: [0, 1]
  • Top-1 Accuracy. The fraction of samples where the predicted class matches the ground truth label.
• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union. Computes the IoU for each class between predicted and ground truth masks, then averages across all classes.
• kNN Accuracy — range: [0, 1]
  • Accuracy of a non-parametric k-Nearest Neighbors classifier (k=20) using frozen representations.
• K-means Accuracy — range: [0, 1]
  • Clustering accuracy obtained by applying K-means to frozen representations and matching cluster labels to ground truth.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for binary classification tasks in sparse probing experiments.

Input / output format

Input: Sentinel-2 multispectral images (12 bands) and/or Sentinel-1 radar images (VV, VH backscatter), typically resized to 264×264 for classification or cropped to 96×96 for segmentation.

Output: Class labels (multi-label for BigEarthNet, single-label for others), segmentation masks, or frozen feature vectors for probing.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'classification':
        if gold.ndim > 1:  # multi-label
            return mean_average_precision(gold, predictions)
        else:
            return accuracy(gold, predictions.argmax(axis=1))
    elif task == 'segmentation':
        return mean_intersection_over_union(gold, predictions)
    elif task == 'knn':
        return knn_accuracy(gold, predictions, k=20)
    elif task == 'clustering':
        return clustering_accuracy(gold, predictions)
    elif task == 'sparse_probing':
        return f1_score(gold, predictions)

Common pitfalls

• Using the full training set instead of the 10% split for BigEarthNet and fMoW-Sentinel, which inflates performance and breaks fair comparison.
• Comparing models trained under different data splits or hyperparameter budgets, as originally reported results vary significantly.
• Ignoring distribution shift when finetuning on datasets like fMoW-Sentinel where the model was pre-trained on the same data, giving baselines an unfair advantage.

Evidence (verbatim from paper)

Table 1: Classification results on four benchmarks, under finetuning (FT), and frozen linear (LP) and nonlinear (MLP) probing. BigEarthNet (10%) mAP, fMoW-Sentinel (10%) Top 1 Acc., EuroSAT Top 1 Acc., Canadian Cropland Top 1 Acc.

Citation

@misc{fuller2023croma,
  title={CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders},
  author={Fuller et al. (2023)},
  year={2023},
  note={arXiv:2311.00566}
}

• arXiv: 2311.00566