Bayesian Optimization Eval

bayesian-optimization-eval

Advancing Bayesian Optimization via Learning Correlated Latent Space — Lee et al. (2023) (arXiv:2310.20258, 2023)

What this evaluates

Evaluates the ability of Bayesian optimization methods to efficiently search discrete spaces (molecules, arithmetic expressions) by maximizing or minimizing a black-box objective function over a limited budget of oracle calls. It probes how well a model aligns its latent representation with the objective landscape to guide search.

Datasets

• Guacamol, TDC DRD3, Arithmetic Expression — total ?; splits: (unstated)

Metrics

• objective value (primary) — range: other
  • The value returned by the black-box function for a generated candidate. Tasks either maximize (e.g., Guacamol MPO) or minimize (e.g., DRD3, arithmetic error) this value. Reported as mean and standard deviation over 3 random seeds across the oracle call budget.
• Pearson correlation — range: [-1, 1]
  • Pearson correlation coefficient between latent space distances (||z_i - z_j||_2) and objective value distances (|y_i - y_j|), measured over the top 10^3 points during optimization.

Input / output format

Input: Candidate discrete structures (molecules in SELFIES format or arithmetic expressions) sampled from a latent space, passed through a decoder to be evaluated by a black-box objective function.

Output: Sequence of candidate structures generated over a budget of oracle calls, along with their corresponding objective values.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    # predictions: list of objective values over oracle calls
    # task_type: 'max' or 'min'
    if task_type == 'max':
        return max(predictions)
    else:
        return min(predictions)

Common pitfalls

• Tasks have different optimization directions (maximization vs minimization); failing to align the metric direction will invert results.
• The 'Score' in ablation tables is recorded at a fixed oracle budget (20k), not the final budget, so comparisons must use consistent oracle counts.
• Results are averaged over 3 repetitions; single-run evaluations will show high variance and may misrank methods.

Evidence (verbatim from paper)

Figure 2, 3 represent the graphs that depict the number of oracle calls, i.e., the number of the black-box function evaluations, and the corresponding mean and standard deviation of objective value.

Citation

@misc{lee2023advancing,
  title={Advancing Bayesian Optimization via Learning Correlated Latent Space},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2310.20258}
}

• arXiv: 2310.20258