Pri Mo Mo Hpo Eval

pri-mo-mo-hpo-eval

Multi-objective Hyperparameter Optimization in the Age of Deep Learning — Basu et al. (2025) (arXiv:2511.08371, 2025)

What this evaluates

Evaluates multi-objective hyperparameter optimization algorithms on deep learning benchmarks, measuring their ability to find high-quality Pareto fronts of validation error and training cost under varying prior conditions and budget constraints.

Datasets

• Yahpo-Gym & PD1 HPO Benchmarks — total ?; splits: test (-1)

Metrics

• mean dominated hypervolume (primary) — range: other
  • Computes the hypervolume indicator (volume of objective space dominated by the found Pareto front relative to a fixed reference point) for each random seed, then averages across 25 seeds.
• relative rankings — range: other
  • Ranks algorithms by their mean dominated hypervolume at each evaluation budget step, then averages the ranks across all benchmarks and seeds to report anytime and final performance.

Input / output format

Input: Hyperparameter configurations sampled by the optimizer, evaluated on deep learning tasks (image classification, language translation) to yield validation error and training cost.

Output: A sequence of evaluated configurations and their corresponding (validation error, training cost) objective pairs, culminating in a final Pareto front.

Scoring recipe

hv_scores = []
for seed in range(25):
    pareto_front = run_optimizer(benchmark, seed, budget=20)
    hv = compute_dominated_hypervolume(pareto_front, reference_point)
    hv_scores.append(hv)
mean_hv = sum(hv_scores) / len(hv_scores)
# Relative ranking: rank algorithms by mean_hv at each budget step, average across benchmarks/seeds

Common pitfalls

• Budget is defined as '20 equivalent full function evaluations'; cheap approximations/low-fidelity runs count fractionally toward the budget, so miscounting distorts the constraint.
• Hypervolume requires a consistent reference point across all benchmarks and seeds; changing it invalidates cross-benchmark comparisons.
• Prior conditions (good/bad/mixed) drastically affect early performance; evaluating only under 'good' priors overstates robustness and masks recovery behavior.

Evidence (verbatim from paper)

We base our multi-objective evaluation on the mean dominated hypervolume across 25 seeds and report relative rankings of the algorithms across budgets. Each optimizer-benchmark-seed combination was run for 20 equivalent full function evaluations, corresponding to typical budgets in practical Deep Learning.

Citation

@misc{basu2025multiobjective,
  title={Multi-objective Hyperparameter Optimization in the Age of Deep Learning},
  author={Basu et al. (2025)},
  year={2025},
  note={arXiv:2511.08371}
}

• arXiv: 2511.08371