Spatiotemporal Forecasting Eval

spatiotemporal-forecasting-eval

Masked Autoregressive Model for Weather Forecasting — Kim et al. (2024) (arXiv:2409.20117, 2024)

What this evaluates

Evaluates spatiotemporal forecasting models on predicting future frames or climate indices from historical observations. It probes pixel-level reconstruction accuracy, structural similarity, and the model's ability to mitigate error propagation over extended lead times.

Datasets

• Moving MNIST — total 20000; splits: train (10000), test (10000)
• TrafficBJ — total 20961; splits: train (19627), test (1334)
• Human 3.6 — total 3759; splits: train (2624), test (1135)
• SEVIR — total 47877; splits: train (35718), test (12159)
• ICAR-ENSO — total 6872; splits: train (5205), test (1667)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted and ground truth frames: (1/N) * Σ(pred - gold)^2.
• MAE — range: other
  • Mean Absolute Error between predicted and ground truth frames: (1/N) * Σ|pred - gold|.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure evaluating luminance, contrast, and structure similarity between image windows.
• CSI-M — range: [0, 1]
  • Critical Success Index averaged over thresholds [16, 74, 133, 160, 181, 219]: CSI = Hits / (Hits + Misses + False Alarms) per threshold.
• C-Nino3.4 — range: [-1, 1]
  • Correlation skill of the three-month-averaged Nino3.4 index (SST anomalies in 170°W-120°W, 5°S-5°N) over 12 forecasting steps.

Input / output format

Input: Spatiotemporal tensor of shape (C, H, W, T) representing past observations (e.g., video frames, radar imagery, or SST anomalies).

Output: Spatiotemporal tensor of shape (C, H, W, T_hat) representing predicted future observations.

Scoring recipe

def compute_metrics(pred, gold):
    mse = np.mean((pred - gold) ** 2)
    mae = np.mean(np.abs(pred - gold))
    ssim = compute_ssim(pred, gold)
    thresholds = [16, 74, 133, 160, 181, 219]
    csi_scores = []
    for thr in thresholds:
        pred_bin = (pred > thr).astype(int)
        gold_bin = (gold > thr).astype(int)
        hits = np.sum((pred_bin == 1) & (gold_bin == 1))
        misses = np.sum((pred_bin == 0) & (gold_bin == 1))
        false_alarms = np.sum((pred_bin == 1) & (gold_bin == 0))
        csi = hits / (hits + misses + false_alarms) if (hits + misses + false_alarms) > 0 else 0
        csi_scores.append(csi)
    csi_m = np.mean(csi_scores)
    return {'MSE': mse, 'MAE': mae, 'SSIM': ssim, 'CSI-M': csi_m}

Common pitfalls

• TrafficBJ metrics (MSE, MAE, SSIM) saturate due to linear past-future relationships, making them poor discriminators for this dataset.
• CSI-M requires averaging over six specific thresholds [16, 74, 133, 160, 181, 219]; using a single threshold or different values yields incorrect scores.
• ENSO evaluation (C-Nino3.4) requires three-month averaging of SST anomalies and correlation skill calculation over 12 forecasting steps, not raw pixel-wise MSE.

Evidence (verbatim from paper)

For the evaluation of common benchmark datasets, we adopt widely used evaluation metrics, including MSE, MAE, Peak Signal to Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM). For rain forecasting models, we use the Critical Success Index (CSI) as an evaluation metric [[38]]. In addition, we validate ENSO forecasting using the Nino SST indices [[16]]. Specifically, the Nino3.4 index represents the averaged SST anomalies across a specific Pacific region (170∘W-120∘W, 5∘S-5∘N), and defines El Niño/La Niña events based on the SST anomalies around the equator.

Citation

@misc{kim2024maskedautoregressive,
  title={Masked Autoregressive Model for Weather Forecasting},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2409.20117}
}

• arXiv: 2409.20117