Uci Gp Benchmark Eval

uci-gp-benchmark-eval

Recommendations for Baselines and Benchmarking Approximate Gaussian Processes — Ober et al. (2024) (arXiv:2402.09849, 2024)

What this evaluates

Evaluates the time-accuracy trade-offs of approximate Gaussian Process regression methods against exact baselines and simple models across multiple UCI regression datasets. It measures how quickly approximations converge to near-exact performance while tracking predictive quality over time.

Datasets

• UCI regression datasets — total ?; splits: train (-1), test (-1)

Metrics

• NLPD (primary) — range: other
  • Negative Log Predictive Density; measures the average negative log-likelihood of the true targets under the predictive distribution.
• RMSE — range: other
  • Root Mean Squared Error; computed as the square root of the mean squared difference between predicted and true targets.
• LML/ELBO — range: other
  • Log Marginal Likelihood or Evidence Lower Bound; measures the model's fit to the training data, with SGPR reporting upper bounds.

Input / output format

Input: Feature vectors and target values from UCI regression datasets.

Output: Predicted means and variances per test point; evaluation reports NLPD, RMSE, and LML/ELBO at specified time checkpoints.

Scoring recipe

def compute_metrics(y_true, y_pred_mean, y_pred_var, model):
    rmse = np.sqrt(np.mean((y_true - y_pred_mean)**2))
    nlpd = -np.mean(0.5 * np.log(2 * np.pi * y_pred_var) + 0.5 * ((y_true - y_pred_mean)**2 / y_pred_var))
    lml = model.log_marginal_likelihood()
    return {'RMSE': rmse, 'NLPD': nlpd, 'LML': lml}

Common pitfalls

• Reporting only final accuracy instead of time-accuracy trade-offs across multiple checkpoints, obscuring convergence behavior.
• Using default solver tolerances or preconditioner sizes for iterative methods, leading to convergence failures or erratic metric spikes.
• Omitting linear regression and constant function baselines, which masks whether the GP approximation actually captures signal.

Evidence (verbatim from paper)

We run each method over 5 different seeds, and additionally use exact GPR baselines, where possible. and additionally use linear regression, constant function, and, where possible, GPR baselines. We plot LML approximations (including upper bounds for SGPR), RMSEs, and NLPDs.

Citation

@misc{ober2024recommendations,
  title={Recommendations for Baselines and Benchmarking Approximate Gaussian Processes},
  author={Ober et al. (2024)},
  year={2024},
  note={arXiv:2402.09849}
}

• arXiv: 2402.09849