The Well Eval

the-well-eval

The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning — Ohana et al. (2024) (arXiv:2412.00568, 2024)

What this evaluates

Evaluates deep learning surrogate models on autoregressive time-series prediction across 16 diverse physics simulations. It probes the model's ability to forecast future spatiotemporal states from short historical snapshots and maintain stability over longer rollout horizons.

Datasets

• The Well — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/PolymathicAI/the_well

Metrics

• VRMSE (primary) — range: other
  • Variance Scaled Root Mean Squared Error. Scaled such that predicting the mean value of the target field yields a score of 1. Computed as RMSE divided by the RMSE of a mean-field baseline, averaged over all physical fields.

Input / output format

Input: A sequence of 4 temporally coarsened, uniformly sampled grid snapshots representing the short history of a physical simulation.

Output: The next single grid snapshot of the simulation.

Scoring recipe

def compute_vrmse(pred, true):
    rmse = np.sqrt(np.mean((pred - true) ** 2))
    mean_pred = np.full_like(true, np.mean(true))
    norm = np.sqrt(np.mean((mean_pred - true) ** 2))
    return rmse / norm

Common pitfalls

• One-step evaluation uses sliding windows sampled from ground truth, while multi-step rollouts always start from the beginning of the simulation, causing performance gaps that reflect physics behavior rather than just model capability.
• NRMSE (using 2-norm normalization) down-weights errors for non-negative fields like pressure or density that have little variation; VRMSE is preferred for these cases.
• Boundary conditions are handled naively according to model defaults, which can unfairly penalize or favor certain architectures (e.g., U-net vs FNO) without a clear trend.

Evidence (verbatim from paper)

Table [2] reports the one-step Variance Scaled Root Mean Squared Error (VRMSE) – defined in Section [E.3] – averaged over all physical fields. We report evaluation on the test set of each model with hyperparameters performing best on the validation set.

Citation

@misc{ohana2024thewell,
  title={The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning},
  author={Ohana et al. (2024)},
  year={2024},
  note={arXiv:2412.00568}
}

• arXiv: 2412.00568