Hyperjump Eval

hyperjump-eval

HyperJump: Accelerating HyperBand via Risk Modelling — Mendes et al. (2021) (arXiv:2108.02479, 2021)

What this evaluates

Evaluates the optimization quality and time efficiency of hyperparameter search algorithms by comparing the test error rate of recommended configurations against wall-clock time across neural architecture and traditional ML benchmarks. It measures how quickly each optimizer converges to near-optimal configurations under sequential and parallel deployment settings.

Datasets

• NATS-Bench — total 15625; splits: test (-1)
• LIBSVM Covertype — total ?; splits: train (-1)

Metrics

• test_error_rate (primary) — range: [0, 1]
  • 1.0 minus the classification accuracy on the held-out test set.
• wall_clock_time — range: seconds
  • Total elapsed seconds from optimization start to configuration recommendation, including training and overhead.

Input / output format

Input: Hyperparameter configuration vectors (e.g., NN topology connections, SVM kernel/gamma/C) and a computational budget (epochs for NATS, training set size for LIBSVM).

Output: Recommended hyperparameter configuration(s) and their corresponding test error rate.

Scoring recipe

def compute_metrics(predictions, gold_labels, start_time, end_time):
    accuracy = sum(predictions == gold_labels) / len(gold_labels)
    test_error_rate = 1.0 - accuracy
    wall_clock_time = end_time - start_time
    return {'test_error_rate': test_error_rate, 'wall_clock_time': wall_clock_time}

Common pitfalls

• Confusing the computational budget (epochs or training set size) with actual wall-clock time; speed-up comparisons must be based on elapsed time, not budget units.
• Assuming the global optimum is known for all benchmarks; only NATS-Bench has an exhaustive evaluation of all 15,625 configurations, while LIBSVM Covertype's optimum is unknown.
• Comparing parallel-only baselines (e.g., ASHA) against sequential methods without adjusting for worker pool size, which skews wall-clock time comparisons.

Evidence (verbatim from paper)

Figure 2 reports the average loss (i.e., the test error rate) and corresponding standard deviation in the shaded areas as a function of the wall clock time (i.e., training and recommendation time).

Citation

@misc{mendes2021hyperjump,
  title={HyperJump: Accelerating HyperBand via Risk Modelling},
  author={Mendes et al. (2021)},
  year={2021},
  note={arXiv:2108.02479}
}

• arXiv: 2108.02479