Satellite Llp Eval

satellite-llp-eval

On-orbit model training for satellite imagery with label proportions — Ramos-Pollán et al. (2023) (arXiv:2306.12461, 2023)

What this evaluates

Evaluates the ability of lightweight deep learning models to predict fine-grained class proportions (e.g., vegetation density, population) from satellite image chips. The protocol measures how well models trained on coarse administrative-level label proportions can recover fine-grained spatial distributions, using both proportion regression and pixel-level segmentation accuracy.

Datasets

• esaworldcover — total ?; splits: test (-1); repo https://github.com/rramosp/llpeo
• humanpop — total ?; splits: test (-1); repo https://github.com/rramosp/llpeo

Metrics

• MAE (primary) — range: [0, 1]
  • Mean Absolute Error computed on the predicted label proportions per chip compared to the ground truth label proportions derived from pixel-level labels.
• F1 — range: [0, 1]
  • Standard F1 score computed from the pixel-level segmentation predictions against ground truth segmentation maps.

Input / output format

Input: Satellite image chips (geospatial raster data).

Output: Pixel-level segmentation maps (class predictions per pixel).

Scoring recipe

for each chip:
  gt_props = count_pixels_by_class(gt_map) / total_pixels
  pred_props = count_pixels_by_class(pred_map) / total_pixels
  mae = mean(abs(pred_props - gt_props))
  f1 = compute_f1_score(pred_map, gt_map)
return mae, f1

Common pitfalls

• Models are trained using only coarse administrative-level label proportions, but evaluated on fine-grained chip-level proportions.
• Simpler models with fewer parameters consistently outperform larger segmentation models on this task, contrary to typical deep learning expectations.
• Evaluation combines both proportion regression (MAE) and pixel segmentation (F1), which may not always correlate.

Evidence (verbatim from paper)

Given the pixel level labels for each chip we compute the label proportions and compare both the segmentation model output (F1) and its proportions (MAE).

Citation

@misc{ramospollan2023onorbit,
  title={On-orbit model training for satellite imagery with label proportions},
  author={Ramos-Pollán et al. (2023)},
  year={2023},
  note={arXiv:2306.12461}
}

• arXiv: 2306.12461