Droughted Eval

droughted-eval

AutoML-Based Drought Forecast with Meteorological Variables — Duan et al. (2022) (arXiv:2207.07012, 2022)

What this evaluates

Evaluates time-series forecasting models on predicting U.S. drought severity across 1 to 6 week horizons using meteorological and static features. It probes both regression accuracy and multi-class classification performance for drought monitoring levels.

Datasets

• DroughtED — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual drought monitor values.
• macro-averaged F1 — range: [0, 1]
  • Macro-averaged F1 score computed over classes 0–5. For regression outputs, predictions are rounded to the nearest integer before computing F1 against rounded gold labels.

Input / output format

Input: Normalized meteorological variables (scaled by median and interquartile range) and static features (scaled by mean and standard deviation), along with target drought monitor values (0–5).

Output: Regression: continuous float value. Classification: discrete integer class (0–5). F1 can also be computed from regression outputs by rounding both predictions and gold labels.

Scoring recipe

def compute_metrics(preds, golds):
    mae = np.mean(np.abs(preds - golds))
    rounded_preds = np.round(preds)
    rounded_golds = np.round(golds)
    f1 = f1_score(rounded_golds, rounded_preds, average='macro')
    return mae, f1

Common pitfalls

• The authors deviate from the original benchmark's validation split (using 60/40 train/val instead of a fixed 1-year validation set), which the authors themselves note makes direct comparison unfair.
• The benchmark originally assessed classification via post-processing of regression outputs, whereas this evaluation trains separate classification models, creating a methodological mismatch.
• F1 scores for regression models are derived by rounding continuous predictions and gold labels, which can artificially inflate or deflate performance compared to direct classification training.

Evidence (verbatim from paper)

The DroughtED benchmark compares F1 score and mean absolute errors (MAE) for Week1 to Week6 forecasts. In this paper, the model is trained in a different manner with the benchmark. Two models are trained: regression (for MAE comparisons) and classification (for F1 comparisons), while the benchmark DL models are trained with a regression loss and the classification performance is assessed with post-processing. In regression, the evaluation metrics is set to MAE, and macro-averaged F1 for classification.

Citation

@misc{duan2022automl,
  title={AutoML-Based Drought Forecast with Meteorological Variables},
  author={Duan et al. (2022)},
  year={2022},
  note={arXiv:2207.07012}
}

• arXiv: 2207.07012