Fg Mae Transfer Eval

fg-mae-transfer-eval

Feature Guided Masked Autoencoder for Self-supervised Learning in Remote Sensing — Yi Wang et al. (arXiv:2310.18653, 2023)

What this evaluates

Evaluates the transfer learning capability of a feature-guided masked autoencoder on remote sensing imagery. It probes the model's ability to adapt to downstream scene classification and semantic segmentation tasks across multispectral and SAR modalities using linear probing and fine-tuning protocols.

Datasets

• BigEarthNet-MM — total ?; splits: train (-1), test (-1)
• BigEarthNet-SAR — total ?; splits: train (-1), test (-1)
• EuroSAT — total ?; splits: train (-1), test (-1)
• EuroSAT-SAR — total ?; splits: train (-1), test (-1)
• DFC2020 — total ?; splits: train (-1), test (-1)

Metrics

• mAP (primary) — range: [0, 1]
  • Mean Average Precision across all classes for multi-label classification.
• F1 — range: [0, 1]
  • Macro-averaged F1 score across all classes.
• OA — range: [0, 1]
  • Overall Accuracy, the ratio of correctly classified pixels or images to the total number.
• AA — range: [0, 1]
  • Average Accuracy, the mean of per-class accuracies.
• mIoU — range: [0, 1]
  • Mean Intersection over Union across all segmentation classes.

Input / output format

Input: Remote sensing image patches (multispectral or SAR) for classification; full-resolution or tiled images for segmentation.

Output: Class labels (single or multi-label) for classification; pixel-wise class assignments for segmentation.

Scoring recipe

# Classification (BigEarthNet, EuroSAT)
preds = model(images)
if multi_label:
    mAP = mean(average_precision(y_true, preds))
    F1 = f1_score(y_true, preds, average='macro')
else:
    OA = accuracy(y_true, preds)
    AA = mean(recall_per_class(y_true, preds))

# Segmentation (DFC2020)
preds = model(images)
IoU = intersection_over_union(preds, targets)
mIoU = mean(IoU)
OA = accuracy(flatten(targets), flatten(preds))
AA = mean(recall_per_class(flatten(targets), flatten(preds)))

Common pitfalls

• Confusing linear probing (frozen encoder, trainable classifier) with fine-tuning (updating encoder weights), as results differ significantly.
• Mixing up multispectral (MS) and SAR modalities, which have different noise characteristics and require different feature targets (HOG+NDI vs HOG).
• Using the 10% subset of BigEarthNet for ablation studies instead of the full 100% set for final benchmarking.

Evidence (verbatim from paper)

Table V demonstrates the transfer learning results on the semantic segmentation dataset DFC2020, where FGMAE-MS outperforms MAE by noticeable margins on all metrics (e.g. 3.4% increase in mIoU).

Citation

@misc{wang2023fgmae,
  title={Feature Guided Masked Autoencoder for Self-supervised Learning in Remote Sensing},
  author={Yi Wang et al.},
  year={2023},
  note={arXiv:2310.18653}
}

• arXiv: 2310.18653