Realpdebench Eval

realpdebench-eval

RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data — Hu et al. (2026) (arXiv:2601.01829, 2026)

What this evaluates

Evaluates scientific machine learning models on sim-to-real transfer for complex physical systems. It probes a model's ability to predict spatiotemporal dynamics from real-world measurements, leveraging simulated pretraining, and assesses long-term prediction stability under autoregressive rollout.

Datasets

• Cylinder — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4Science-WestlakeU/RealPDEBench
• ControlledCylinder — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4Science-WestlakeU/RealPDEBench
• FSI — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4Science-WestlakeU/RealPDEBench
• Foil — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4Science-WestlakeU/RealPDEBench
• Combustion — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4Science-WestlakeU/RealPDEBench

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error between predicted and ground truth spatiotemporal fields.
• Rel L2 — range: [0, 1]
  • Relative L2 Error, computed as the L2 norm of the prediction error divided by the L2 norm of the ground truth.
• fRMSE — range: other
  • Frequency RMSE, measuring prediction error in the frequency domain to capture global periodicity and dynamics.
• Update Ratio — range: [0, 1]
  • Ratio of update iterations required for real-world finetuning to reach optimal performance compared to direct real-world training.

Input / output format

Input: Spatiotemporal PDE data consisting of input and output fields over T time steps. Simulated inputs may contain unmeasured modalities that are randomly masked during training.

Output: Predicted spatiotemporal fields over T time steps (or NT steps for autoregressive evaluation).

Scoring recipe

def compute_metrics(pred, gold):
    rmse = np.sqrt(np.mean((pred - gold) ** 2))
    rel_l2 = np.linalg.norm(pred - gold) / np.linalg.norm(gold)
    f_pred = np.fft.fft(pred)
    f_gold = np.fft.fft(gold)
    f_rmse = np.sqrt(np.mean(np.abs(f_pred - f_gold) ** 2))
    return rmse, rel_l2, f_rmse

Common pitfalls

• Data splits are performed at the parameter level, not trajectory level, meaning different physical configurations are held out for testing.
• Autoregressive evaluation compounds error rapidly; single-round and multi-round (e.g., 10-round) results can differ drastically, requiring careful reporting of rollout steps.
• Simulated data contains extra modalities not present in real-world measurements; models must be trained with random masking of these unmeasured modalities to avoid unfair advantages.

Evidence (verbatim from paper)

We report the RMSE, Relative L2 Error, fRMSE, and Update Ratio of all datasets and baselines under the three training categories, as summarized in Table 1.

Citation

@misc{hu2026realpdebench,
  title={RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2601.01829}
}

• arXiv: 2601.01829