Qoblib Eval

qoblib-eval

Quantum Optimization Benchmarking Library - The Intractable Decathlon — Koch et al. (2025) (arXiv:2504.03832, 2025)

What this evaluates

Evaluates the performance of quantum and classical optimization algorithms on intractable combinatorial problems. It measures solution quality, algorithmic success rates, and computational efficiency to track progress toward quantum advantage.

Datasets

• QOBLIB — total ?; splits: benchmark (-1)

Metrics

• best_objective_value (primary) — range: other
  • The minimum (for minimization) or maximum (for maximization) objective function value found by the algorithm across all repetitions.
• success_rate — range: [0, 1]
  • Fraction of runs yielding a feasible solution within threshold ε of the best found value: ≤ (1+ε)*fmin or ≥ (1-ε)*fmax.
• time_to_solution — range: other
  • τ log(1 - 0.99) / log(1 - p_gs), where τ is execution time per shot and p_gs is the probability of sampling the optimal solution.
• total_runtime — range: other
  • Average wall-clock time across all algorithm repetitions, excluding hardware queuing times.

Input / output format

Input: Problem instance specification including modeling approach (e.g., QUBO, HUBO, ILP), number and type of decision variables, coefficient ranges, and constraints.

Output: Solution bitstring/vector, corresponding objective value, feasibility status, and runtime breakdown (pre-processing, QPU/CPU/GPU, post-processing).

Scoring recipe

def compute_metrics(predictions, threshold_eps=0.0):
    best_obj = min(p['objective'] for p in predictions)
    successful = sum(1 for p in predictions if p['feasible'] and p['objective'] <= (1 + threshold_eps) * best_obj)
    success_rate = successful / len(predictions)
    avg_runtime = sum(p['total_runtime'] for p in predictions) / len(predictions)
    return {'best_objective_value': best_obj, 'success_rate': success_rate, 'total_runtime': avg_runtime}

Common pitfalls

• Runtimes must exclude hardware queuing times; only actual execution/pre/post-processing time should be reported.
• Stochastic algorithms require multiple repetitions; metrics like success rate and runtime must be averaged across runs, not reported per single trial.
• Optimality bounds must be provided if available; otherwise marked N/A, as they are critical for comparing solution quality across different algorithms.

Evidence (verbatim from paper)

The metrics were chosen to enable a fair and systematic comparison of solutions and algorithms. More specifically, we ask for the identification of the concrete problem instance and the submitter(s). ... Furthermore, the best objective value found (for optimization problems) as well as the corresponding solution (in addition to the template) should be given. ... the number of successful runs that result in feasible solutions or return solutions close to the best found solutions (cf. Table 10 for more details) should be reported. ... As discussed in Section 3, the runtimes should not include potential queuing times for hardware access.

Citation

@misc{koch2025qoblib,
  title={Quantum Optimization Benchmarking Library - The Intractable Decathlon},
  author={Koch et al. (2025)},
  year={2025},
  note={arXiv:2504.03832}
}

• arXiv: 2504.03832