Noaa Sst Forecasting Eval

noaa-sst-forecasting-eval

Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods — Hamzi et al. (2021) (arXiv:2103.10935, 2021)

What this evaluates

Evaluates the ability of data-driven models to forecast low-dimensional geophysical dynamics (sea surface temperature and air temperature) from historical time-series observations. It probes long-horizon prediction accuracy, bias-variance trade-offs, and computational efficiency compared to physics-based and deep learning baselines.

Datasets

• NOAA-SST — total 1914; splits: train (427), test (1487)
• NOAA-NCEP NAM — total ?; splits: train (-1), test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error between predicted and observed values, measured in degrees Celsius. Calculated as the square root of the mean of squared differences over the testing period.
• Correlation coefficient — range: [-1, 1]
  • Pearson correlation coefficient measuring linear relationship between forecast and truth across the spatial domain.
• Cosine similarity — range: [0, 1]
  • Cosine similarity between forecast and truth vectors, used to assess the ability to detect extreme fluctuations.

Input / output format

Input: Time-delayed input windows of historical geophysical observations (e.g., 7-day sequences) projected onto a low-dimensional Proper Orthogonal Decomposition (POD) basis or used directly.

Output: Forecasted values for a specified future horizon (e.g., 8 weeks or 7 days ahead), either as POD coefficients or direct temperature/field values.

Scoring recipe

def compute_rmse(pred, true):
    return np.sqrt(np.mean((pred - true) ** 2))

def compute_correlation(pred, true):
    return np.corrcoef(pred.flatten(), true.flatten())[0, 1]

def compute_cosine_similarity(pred, true):
    dot = np.dot(pred.flatten(), true.flatten())
    norm = np.linalg.norm(pred) * np.linalg.norm(true)
    return dot / norm if norm > 0 else 0.0

Common pitfalls

• The evaluation relies on time-delayed input windows rather than single-step predictions, requiring careful alignment of temporal strides between input sequences and target horizons.
• Computational cost baselines (e.g., LSTM) often exclude neural architecture search overhead, making direct wall-time comparisons unfair unless search costs are explicitly accounted for.
• Error characteristics differ significantly between datasets: NOAA-SST exhibits bias-dominated errors (low noise), while NOAA-NCEP NAM shows variance-dominated errors (high noise), which can mislead model selection if not analyzed separately.

Evidence (verbatim from paper)

The reconstruction accuracy from the forecast is compared in a series of assessments beginning with root-mean-squared error (RMSE) assessments as shown in Figure 8 for the testing time period. POD-RKHS is seen to provide competitive results in comparison to persistence and climatology at the lower latitudes.

Citation

@misc{hamzi2021data,
  title={Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods},
  author={Hamzi et al. (2021)},
  year={2021},
  note={arXiv:2103.10935}
}

• arXiv: 2103.10935