Piano Pde Weather Eval

piano-pde-weather-eval

PIANO: Physics Informed Autoregressive Network — Nagda et al. (2025) (arXiv:2508.16235, 2025)

What this evaluates

Evaluates the capability of physics-informed autoregressive models to accurately forecast time-dependent partial differential equations and global atmospheric variables over multi-step horizons.

Datasets

• PDE Benchmarks (Wave, Reaction, Convection, Heat) — total ?; splits: test (-1)
• ERA5 — total ?; splits: test (-1)

Metrics

• rMAE — range: percent
  • Relative Mean Absolute Error: the mean absolute difference between predicted and ground truth states, normalized by the magnitude of the ground truth.
• rRMSE — range: percent
  • Relative Root Mean Squared Error: the square root of the mean squared difference between predictions and ground truth, normalized by the ground truth magnitude.
• RMSE (primary) — range: other
  • Root Mean Squared Error: the square root of the average squared differences between predicted and observed values. For weather forecasting, it is latitude-weighted to account for spherical geometry.
• ACC — range: [-1, 1]
  • Anomaly Correlation Coefficient: measures the correlation between predicted and observed anomalies (deviations from the mean), capturing directional accuracy.

Input / output format

Input: Discretized spatio-temporal state grids at initial or previous time steps, spatial gradients, velocity fields, and spatiotemporal embeddings.

Output: Predicted spatio-temporal state u(x,t) and velocity field v(t) at the target time step.

Scoring recipe

def compute_metrics(pred, true, lat):
    # PDE relative errors
    rMAE = mean(abs(pred - true)) / mean(abs(true))
    rRMSE = sqrt(mean((pred - true)**2)) / sqrt(mean(true**2))
    
    # Weather metrics (latitude-weighted)
    weights = cos(lat) / mean(cos(lat))
    RMSE = sqrt(mean(weights * (pred - true)**2))
    ACC = cov(pred - mean(pred), true - mean(true)) / (std(pred) * std(true))
    return rMAE, rRMSE, RMSE, ACC

Common pitfalls

• Baseline papers do not report variance, so statistical significance testing was not performed.
• Weather metrics require latitude-weighting to correctly account for the spherical geometry of the Earth.
• PDE benchmarks use relative errors (normalized by ground truth), not absolute errors, which changes how performance scales with solution magnitude.

Evidence (verbatim from paper)

Performance is measured using relative Mean Absolute Error (rMAE) and relative Root Mean Squared Error (rRMSE), which are standard metrics in the PINN literature (Xu et al. [2025]). ... Performance is evaluated using two standard metrics: root mean square error (RMSE) and anomaly correlation coefficient (ACC). RMSE quantifies the absolute prediction error, while ACC measures the correlation between predicted and observed anomalies, capturing the directional accuracy. Both metrics are latitude-weighted to reflect the spherical geometry of the Earth.

Citation

@misc{nagda2025piano,
  title={PIANO: Physics Informed Autoregressive Network},
  author={Nagda et al. (2025)},
  year={2025},
  note={arXiv:2508.16235}
}

• arXiv: 2508.16235