Cl Engineering Regression Eval

cl-engineering-regression-eval

Continual Learning Strategies for 3D Engineering Regression Problems: A Benchmarking Study — Samuel et al. (2025) (arXiv:2504.12503, 2025)

What this evaluates

Evaluates continual learning strategies for 3D engineering regression tasks, measuring their ability to learn from sequential data streams while mitigating catastrophic forgetting and maintaining predictive accuracy across parametric and point cloud modalities.

Datasets

• SplitSHIPD-Par — total ?; splits: test (-1)
• SplitSHIPD-PC — total ?; splits: test (-1)
• SplitSHAPENET — total ?; splits: test (-1)
• SplitRAADL — total ?; splits: test (-1)
• SplitDRIVAERNET — total ?; splits: test (-1)
• SplitDRIVAERNET++-Par — total ?; splits: test (-1)
• SplitDRIVAERNET++-PC — total ?; splits: test (-1)

Metrics

• MPE (primary) — range: percent
  • Mean Percent Error: the average of the absolute percentage difference between predicted and true values across the test set. Lower is better.
• Forgetting Ratio — range: other
  • Measures the drop in performance on previously learned experiences after training on new ones. Calculated as the difference in error between the model's performance immediately after learning an experience and its final performance. Negative values indicate positive transfer.
• MAE — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and true values. Reported in units of the target variable (e.g., ×10⁻³).

Input / output format

Input: 3D engineering data in parametric or point cloud modalities, presented sequentially as discrete experiences in a stream.

Output: Continuous regression target value corresponding to the engineering property.

Scoring recipe

def compute_mpe(y_true, y_pred):
    return np.mean(np.abs(y_pred - y_true) / y_true) * 100

def compute_forgetting_ratio(error_at_experience, error_at_final):
    return (error_at_experience - error_at_final) / error_at_experience

Common pitfalls

• Negative forgetting ratios are valid and indicate positive transfer (improvement on past data), not a calculation error.
• MPE and MAE are reported in different scales across datasets; direct cross-dataset comparison requires normalization.
• Strategies are evaluated under bin-incremental and input-incremental scenarios, meaning experience boundaries and input distributions shift, affecting baseline comparability.

Evidence (verbatim from paper)

We evaluated three continual learning strategies (Replay, GEM, EWC) across various engineering datasets, input modalities (parametric and point cloud), and continual learning scenarios (bin incremental and input incremental). For each benchmark, the three strategies are compared based on each strategy’s final MPE and average forgetting ratio for the full test set (Tables[2] and [10]), their MAEs and forgetting ratios for individual experiences (Tables[3]-[9] and Tables[11] and [12]), and their incremental MAE and forgetting on experiences it encountered within the stream (Figure[6]).

Citation

@misc{samuel2025continual,
  title={Continual Learning Strategies for 3D Engineering Regression Problems: A Benchmarking Study},
  author={Samuel et al. (2025)},
  year={2025},
  note={arXiv:2504.12503}
}

• arXiv: 2504.12503