Eurosat Eval

eurosat-eval

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification — Helber et al. (2017) (arXiv:1709.00029, 2017)

What this evaluates

This benchmark evaluates a model's ability to classify land use and land cover types from multi-spectral satellite imagery. It probes the capability to distinguish between 10 distinct environmental classes using patch-based remote sensing inputs across different spectral band configurations.

Datasets

• EuroSAT — total 27000; splits: train (-1), test (-1); repo https://github.com/phelber/eurosat

Metrics

• classification accuracy (%) (primary) — range: percent
  • The percentage of correctly classified image patches out of the total number of patches in the test set. Calculated as (correct predictions / total predictions) * 100.

Input / output format

Input: Satellite image patches provided as RGB color space images (combining bands B04, B03, B02), single-band images (replicated across three input channels), or shortwave-infrared/color-infrared band combinations.

Output: A single class label from the 10 predefined land use/land cover categories.

Scoring recipe

correct = 0
total = 0
for patch, true_label in test_dataset:
    pred_label = model(patch)
    if pred_label == true_label:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• The train/test split must be applied class-wise to maintain balanced class distributions across splits.
• Pretraining the CNN backbone on ILSVRC-2012 (ImageNet) is required to achieve reported state-of-the-art results; random initialization yields approximately 2% lower accuracy.
• Single-band inputs are fed to 3-channel networks by replicating the same band across all channels, which differs from true multi-spectral processing and affects performance comparisons.

Evidence (verbatim from paper)

We respectively split each dataset in a training and a test set (80/20 ratio). We ensured that the split is applied class-wise. While the red, green and blue bands are covered by almost all aerial and satellite image datasets, the proposed EuroSAT dataset consists of 13 spectral bands. For the comparative evaluation, we computed images in the RGB color space combining the bands red (B04), green (B03) and blue (B02). For benchmarking, we evaluated the performance of the Bag-of-Visual-Words (BoVW) approach using SIFT features and a trained SVM. In addition, we trained a shallow Convolutional Neural Network (CNN), a ResNet-50 and a GoogleNet model on the training set. We calculated the overall classification accuracy to evaluate the performance of the different models on the considered datasets.

Citation

@misc{helber2017eurosat,
  title={EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification},
  author={Helber et al. (2017)},
  year={2017},
  note={arXiv:1709.00029}
}

• arXiv: 1709.00029