Methane Segmentation Eval

methane-segmentation-eval

Optimizing Methane Detection On Board Satellites: Speed, Accuracy, and Low-Power Solutions for Resource-Constrained Hardware — Herec et al. (2025) (arXiv:2507.01472, 2025)

What this evaluates

Evaluates the capability of hyperspectral image processing models to detect and segment methane plumes on resource-constrained satellite hardware. It probes the trade-off between detection accuracy (precision, recall, F1) and computational efficiency (runtime) across different spectral enhancement filters and lightweight neural networks.

Datasets

• STARCOP — total ?; splits: test (-1); repo https://github.com/zaitra/methane-filters-benchmark

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision is TP/(TP+FP), Recall is TP/(TP+FN).
• F1 - Strong — range: [0, 1]
  • F1 score computed exclusively on instances or tiles annotated as strong methane plumes.

Input / output format

Input: 512x512 pixel tiles of hyperspectral imagery (72 channels) concatenated with RGB bands.

Output: Binary segmentation mask indicating methane plume presence per pixel.

Scoring recipe

def compute_f1(pred_mask, gold_mask):
    tp = np.sum((pred_mask == 1) & (gold_mask == 1))
    fp = np.sum((pred_mask == 1) & (gold_mask == 0))
    fn = np.sum((pred_mask == 0) & (gold_mask == 1))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
# For F1 - Strong, apply compute_f1 only to regions annotated as strong plumes.

Common pitfalls

• Original Mag1c runtime is measured on 512x512 tiles but the STARCOP dataset provides precomputed column-wise scene products, making its reported runtime an aspirational upper bound rather than a strict baseline.
• Metrics are averaged over 5 repeated training runs with standard deviations reported; ignoring variance may misrepresent model stability.
• F1 - Strong isolates performance on strong plumes only, potentially masking poor detection on weak or medium plumes.

Evidence (verbatim from paper)

The original Mag1c method achieves the highest F1 score for strong plumes (67.50 %) but has a very long runtime (109.61 s). TABLE I: Results showing runtimes and metrics for 512x512 tiles with 72 channels. The total runtime is determined by adding the inference time to the optimized runtime (or, if an optimization was not implemented, the original runtime). For entries that do not mention an ML model, the morphological baseline was applied for inference. The average score and standard deviation are shown for 5 repeated training runs.

Citation

@misc{herec2025methane,
  title={Optimizing Methane Detection On Board Satellites: Speed, Accuracy, and Low-Power Solutions for Resource-Constrained Hardware},
  author={Herec et al. (2025)},
  year={2025},
  note={arXiv:2507.01472}
}

• arXiv: 2507.01472