ml4cfd-competition-eval
NeurIPS 2024 ML4CFD Competition: Results and Retrospective Analysis — Yagoubi et al. (2024) (NeurIPS 2024 / arXiv:2506.08516, 2024)
What this evaluates
Evaluates machine learning surrogates for 2D airfoil aerodynamics on prediction accuracy, computational speed-up, physical consistency, and out-of-distribution generalization. The benchmark compares learned models against a standard CFD solver (OpenFOAM) across in-distribution and novel geometric configurations.
Datasets
Metrics
Global Score (primary) — range: percent
- Weighted aggregation of ML performance (40%), physical consistency (30%), and OOD generalization (30%). ML score combines accuracy (75%) and speed-up (25%). OOD score combines OOD accuracy (42%), OOD physics (33%), and OOD speed-up (25%).
Accuracy — range: other
- Mean error across flow field variables ($\overline{u}{x}$, $\overline{u}{y}$, $\overline{p}$, $\overline{\nu}{t}$, $\overline{p}{s}$). Lower values indicate better performance.
Speed-up — range: other
- Ratio of CFD solver runtime to model inference runtime. Higher values indicate faster prediction.
Physical Criteria — range: other
- Evaluation of integral aerodynamic coefficients ($C_D$, $C_L$) and their correlation coefficients ($\rho_D$, $\rho_L$) against ground truth.
Input / output format
Input: Airfoil geometry representation (e.g., mesh coordinates, latent embeddings, or surface parameters) and flow conditions (e.g., angle of attack, Reynolds number).
Output: Predicted flow field variables ($\overline{u}{x}$, $\overline{u}{y}$, $\overline{p}$, $\overline{\nu}{t}$, $\overline{p}{s}$) and integral aerodynamic coefficients ($C_D$, $C_L$).
Scoring recipe
# Weights from Table 1
ml_weight = 0.40
physics_weight = 0.30
ood_weight = 0.30
ml_score = (0.75 * accuracy_metric) + (0.25 * speedup_metric)
ood_score = (0.42 * ood_accuracy) + (0.33 * ood_physics) + (0.25 * ood_speedup)
global_score = (ml_weight * ml_score) + (physics_weight * physics_criteria_score) + (ood_weight * ood_score)
return global_score
Common pitfalls
- Speed-up is heavily downweighted (25% in ML, 25% in OOD) compared to accuracy, so fast but inaccurate models rank poorly.
- Physical consistency ($C_D$, $C_L$, correlation coefficients) is evaluated separately from field accuracy, requiring models to preserve integral conservation laws.
- OOD evaluation uses distinct geometric configurations not seen during training, testing extrapolation rather than interpolation.
Evidence (verbatim from paper)
Rankings were determined based on a global score that aggregated performance across several key criteria, including machine learning (accuracy and speed-up), physics, and out-of-distribution (OOD) generalization.
Citation
@misc{yagoubi2024ml4cfd,
title={NeurIPS 2024 ML4CFD Competition: Results and Retrospective Analysis},
author={Yagoubi et al. (2024)},
year={2024},
note={NeurIPS 2024 / arXiv:2506.08516}
}
1---2name: ml4cfd-competition-eval3description: Evaluates machine learning surrogates for 2D airfoil aerodynamics on prediction accuracy, computational speed-up, physical consistency, and out-of-distribution generalization. The benchmark compares learned models against a standard CFD solver (OpenFOAM) across in-distribution and novel geometric configurations. Use when the user wants to benchmark on ML4CFD Competition Dataset, or asks about evaluating this task. Reports Global Score.4---56# ml4cfd-competition-eval78> NeurIPS 2024 ML4CFD Competition: Results and Retrospective Analysis — Yagoubi et al. (2024) (NeurIPS 2024 / arXiv:2506.08516, 2024)910## What this evaluates1112Evaluates machine learning surrogates for 2D airfoil aerodynamics on prediction accuracy, computational speed-up, physical consistency, and out-of-distribution generalization. The benchmark compares learned models against a standard CFD solver (OpenFOAM) across in-distribution and novel geometric configurations.1314## Datasets1516- **ML4CFD Competition Dataset** — total ?; splits: in-distribution (-1), out-of-distribution (-1); repo https://github.com/IRT-SystemX/NeurIPS2024-ML4CFD-competition-Starting-Kit1718## Metrics1920- `Global Score` **(primary)** — range: percent21 - Weighted aggregation of ML performance (40%), physical consistency (30%), and OOD generalization (30%). ML score combines accuracy (75%) and speed-up (25%). OOD score combines OOD accuracy (42%), OOD physics (33%), and OOD speed-up (25%).22- `Accuracy` — range: other23 - Mean error across flow field variables ($\overline{u}_{x}$, $\overline{u}_{y}$, $\overline{p}$, $\overline{\nu}_{t}$, $\overline{p}_{s}$). Lower values indicate better performance.24- `Speed-up` — range: other25 - Ratio of CFD solver runtime to model inference runtime. Higher values indicate faster prediction.26- `Physical Criteria` — range: other27 - Evaluation of integral aerodynamic coefficients ($C_D$, $C_L$) and their correlation coefficients ($\rho_D$, $\rho_L$) against ground truth.2829## Input / output format3031**Input**: Airfoil geometry representation (e.g., mesh coordinates, latent embeddings, or surface parameters) and flow conditions (e.g., angle of attack, Reynolds number).3233**Output**: Predicted flow field variables ($\overline{u}_{x}$, $\overline{u}_{y}$, $\overline{p}$, $\overline{\nu}_{t}$, $\overline{p}_{s}$) and integral aerodynamic coefficients ($C_D$, $C_L$).3435## Scoring recipe3637```python38# Weights from Table 139ml_weight = 0.4040physics_weight = 0.3041ood_weight = 0.304243ml_score = (0.75 * accuracy_metric) + (0.25 * speedup_metric)44ood_score = (0.42 * ood_accuracy) + (0.33 * ood_physics) + (0.25 * ood_speedup)4546global_score = (ml_weight * ml_score) + (physics_weight * physics_criteria_score) + (ood_weight * ood_score)47return global_score48```4950## Common pitfalls5152- Speed-up is heavily downweighted (25% in ML, 25% in OOD) compared to accuracy, so fast but inaccurate models rank poorly.53- Physical consistency ($C_D$, $C_L$, correlation coefficients) is evaluated separately from field accuracy, requiring models to preserve integral conservation laws.54- OOD evaluation uses distinct geometric configurations not seen during training, testing extrapolation rather than interpolation.5556## Evidence (verbatim from paper)5758> Rankings were determined based on a global score that aggregated performance across several key criteria, including machine learning (accuracy and speed-up), physics, and out-of-distribution (OOD) generalization.5960## Citation6162```bibtex63@misc{yagoubi2024ml4cfd,64 title={NeurIPS 2024 ML4CFD Competition: Results and Retrospective Analysis},65 author={Yagoubi et al. (2024)},66 year={2024},67 note={NeurIPS 2024 / arXiv:2506.08516}68}69```7071- arXiv: 2506.08516
qhjqhj00