Droughtset Eval

droughtset-eval

DroughtSet: Understanding Drought Through Spatial-Temporal Learning — Xuwei Tan et al. (2024) (arXiv:2412.15075, 2024)

What this evaluates

Evaluates spatiotemporal forecasting models on predicting three drought indices (soil moisture, evaporative stress index, and solar-induced chlorophyll fluorescence) across the U.S. CONUS using weekly climate and vegetation data. It also assesses the models' ability to classify drought events based on soil moisture percentiles.

Datasets

• DroughtSet — total ?; splits: train (380801), test (93220); repo https://github.com/osu-srml/DroughtSet

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error averaged over the 26-week prediction period and all test pixels. Values in the paper are reported multiplied by 10^-3.
• Accuracy — range: percent
  • Percentage of correctly classified drought vs. non-drought weeks, where drought is defined as soil moisture falling below the 30th percentile threshold.
• Precision — range: percent
  • Ratio of correctly predicted drought weeks to all weeks predicted as drought.

Input / output format

Input: Weekly spatiotemporal features (climate, physical, vegetation) and static variables (e.g., land cover) for a 100-week historical window per pixel, normalized to [0,1] with NaNs imputed by yearly averages.

Output: Predicted values for the subsequent 26-week period for three drought indices: Soil Moisture, Evaporative Stress Index (ESI), and Solar-induced Chlorophyll Fluorescence (SIF).

Scoring recipe

def compute_metrics(y_true, y_pred, threshold_percentile=30):
    mae = np.mean(np.abs(y_pred - y_true))
    threshold = np.percentile(y_true, threshold_percentile)
    y_true_drought = (y_true < threshold).astype(int)
    y_pred_drought = (y_pred < threshold).astype(int)
    accuracy = np.mean(y_true_drought == y_pred_drought)
    tp = np.sum((y_pred_drought == 1) & (y_true_drought == 1))
    fp = np.sum((y_pred_drought == 1) & (y_true_drought == 0))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    return mae, accuracy, precision

Common pitfalls

• The dataset uses overlapping 100-week training and 26-week prediction windows slid by 26 weeks, which requires careful handling to avoid temporal data leakage during evaluation.
• All features and targets are normalized by their maximum values to [0,1] before training, so predictions must be rescaled to original units for meaningful MAE comparison.
• Drought classification relies on a fixed 30th percentile threshold derived from historical data, not a dynamic or rolling threshold.

Evidence (verbatim from paper)

We split the pixels by 5x5 pixel block to avoid similar neighboring pixels and randomly select 80% blocks as training pixels and the remaining 20% for testing. Each window consists of 100 weeks (approximately 2 years) designated as the training period, followed by 26 weeks (approximately half a year) designated as the prediction period. The mean absolute error is used as the loss function. We use the 30th percentile as the threshold in our analysis. Soil moisture values below this percentile are considered as soil moisture drought. Drought is compared with other methods in terms of accuracy and precision.

Citation

@misc{tan2024droughtset,
  title={DroughtSet: Understanding Drought Through Spatial-Temporal Learning},
  author={Xuwei Tan et al. (2024)},
  year={2024},
  note={arXiv:2412.15075}
}

• arXiv: 2412.15075