Bounded Nbeddyn Eval

bounded-nbeddyn-eval

Bounded nonlinear forecasts of partially observed geophysical systems with physics-constrained deep learning — Said Ouala et al. (arXiv:2202.05750, 2022)

What this evaluates

This evaluation probes a model's ability to forecast and reconstruct the dynamics of partially observed, chaotic geophysical systems. It specifically tests short-term prediction accuracy and long-term topological stability (boundedness) under both in-distribution and out-of-attractor initial conditions.

Datasets

• Lorenz-63 — total 5000; splits: train (4000), test (1000); repo https://github.com/CIA-Oceanix/Bounded-NbedDyn
• Lorenz-96 — total 5000; splits: train (4000), test (1000); repo https://github.com/CIA-Oceanix/Bounded-NbedDyn

Metrics

• RMSE (primary) — range: other
  • Root mean square error between predicted and ground-truth state variables over the test set.
• Largest Lyapunov Exponent — range: other
  • Computed from a long-term simulation (10,000 time steps) of the predicted trajectory to measure chaotic divergence and topological fidelity.

Input / output format

Input: Sequential partial observations of the system (e.g., first variable for Lorenz-63, first 20 states for Lorenz-96) sampled at dt=0.01. Benchmarks also use delay-embedding representations of these observations.

Output: Forecasted state sequences for the full system (or observed subset) over future time steps, evaluated at t0+dt and t0+4dt for short-term, and up to 10,000 steps for long-term stability.

Scoring recipe

def compute_rmse(pred, true):
    return np.sqrt(np.mean((pred - true) ** 2))

def compute_lyapunov(traj, dt=0.01, steps=10000):
    # Compute largest Lyapunov exponent using standard algorithm (e.g., Rosenstein)
    # on the first `steps` time steps of the predicted trajectory
    return lyapunov_exponent(traj[:steps], dt)

Common pitfalls

• Unconstrained models often diverge to infinity when tested with initial conditions outside the training attractor, making long-term metrics undefined.
• Benchmark baselines (SR, RNN, Latent-ODE) are highly sensitive to delay-embedding hyperparameters (lag tau, dimension d_E), which must be carefully tuned via mutual information or false nearest neighbors.
• The Lyapunov exponent requires a sufficiently long simulation (10,000 steps) and is only meaningful if the trajectory remains bounded.

Evidence (verbatim from paper)

We evaluate in Table 1 the short-term forecast as well as the topological structure of the simulated limit-sets (illustrated for instance through the largest Lyapunov exponent). Regarding the short-term forecast, both versions of the NbedDyn model outperform benchmarked state-of-the-art techniques in terms of root mean square error (RMSE).

Citation

@misc{ouala2022bounded,
  title={Bounded nonlinear forecasts of partially observed geophysical systems with physics-constrained deep learning},
  author={Said Ouala et al.},
  year={2022},
  note={arXiv:2202.05750}
}

• arXiv: 2202.05750