Neural Operator Comparison Eval

neural-operator-comparison-eval

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data — Lu et al. (2021) (arXiv:2111.05512, 2021)

What this evaluates

Evaluates the accuracy and robustness of neural operator models (DeepONet, FNO, and variants) in learning mappings between function spaces for solving partial differential equations across various physical domains and geometries.

Datasets

• PDE Operator Benchmark Suite (Lu et al. 2021) — total ?; splits: train (-1), test (-1)

Metrics

• L2 relative error (primary) — range: percent
  • Computed as the L2 norm of the difference between predicted and ground truth fields divided by the L2 norm of the ground truth field. Reported as a percentage. Mean and standard deviation are calculated over five independent training trials.

Input / output format

Input: Discretized function or field data representing inputs to the PDE operator (e.g., initial conditions, permeability fields, boundary conditions, or electric potential fields) on a fixed grid or mesh.

Output: Discretized predicted function or field data representing the solution to the PDE operator (e.g., solution fields, pressure fields, or concentration fields) on the same grid or mesh.

Scoring recipe

def compute_l2_relative_error(pred, true):
    return np.linalg.norm(pred - true) / np.linalg.norm(true)

errors = []
for trial in range(5):
    pred = model(input)
    errors.append(compute_l2_relative_error(pred, true) * 100)
mean_error = np.mean(errors)
std_error = np.std(errors)
return mean_error, std_error

Common pitfalls

• FNO performance degrades severely with complex geometries and noisy data unless modified (e.g., dgFNO+).
• Output normalization significantly impacts FNO accuracy but has minimal effect on DeepONet.
• Grid resolution and mesh structure must be handled carefully; FNO assumes regular grids while DeepONet handles irregular domains better.

Evidence (verbatim from paper)

We compare the performance of DeepONet and FNO on 16 different problems listed in Table 2. To evaluate the performance of the networks, we compute the $L^2$ relative error of the predictions, and for each case, five independent training trials are performed to compute the mean error and the standard deviation.

Citation

@misc{lu2021neuraloperatorcomparison,
  title={A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data},
  author={Lu et al. (2021)},
  year={2021},
  note={arXiv:2111.05512}
}

• arXiv: 2111.05512