Surrogate Genetic Recommender Eval

surrogate-genetic-recommender-eval

Benchmarking Surrogate-Assisted Genetic Recommender Systems — Gabor et al. (2019) (arXiv:1908.02880, 2019)

What this evaluates

Probes the ability of surrogate-assisted genetic algorithms to optimize continuous 2D functions under tight evaluation budgets, simulating interactive recommendation scenarios where user feedback is sparse and dynamic.

Datasets

• Bohachevsky, Ackley, and Schwefel benchmark functions — total ?; splits: test (-1)

Metrics

• Best Fitness (primary) — range: other
  • The minimum objective function value achieved across all evaluated candidate solutions. Lower values indicate better optimization performance, with the global optimum strictly at 0.

Input / output format

Input: 2D real-valued vectors (x ∈ ℝ²) representing candidate solutions in the search space.

Output: Iteratively generated recommendations (2D vectors) and their corresponding fitness evaluations, updated based on surrogate model predictions and simulated user acceptance.

Scoring recipe

def score(predictions, gold):
    # predictions: list of (x, f(x)) evaluated in chronological order
    best_fitness = min(f for _, f in predictions)
    
    accepted = 0
    worst_so_far = float('inf')
    for _, f in predictions:
        if f < worst_so_far:
            worst_so_far = f
        # A suggestion is accepted if its fitness is better than the worst evaluated so far
        if f < worst_so_far:
            accepted += 1
            
    accepted_rate = accepted / len(predictions) if predictions else 0.0
    return best_fitness, accepted_rate

Common pitfalls

• Benchmarks are synthetic 2D mathematical functions, not real-world user-item interaction data.
• The 'accepted suggestions' metric is relative to the current worst evaluated item in the sequence, not a fixed absolute threshold.
• Results are averaged over 10 stochastic repetitions to mitigate outlier influence, but individual runs may vary significantly due to the optimization process.

Evidence (verbatim from paper)

In order to run a multitude of tests on the performance of the employed approach and the employed models, we opt for standard evolutionary benchmark functions instead of real human interaction. We used the implementation for Bohachevsky, Ackley, and Schwefel functions... All of these are constrained to a specific subset of R^2 and are to be minimized with a best fitness value of 0. The number of accepted suggestions is obtained by counting the suggestions that are evaluated better than the worst evaluated item at that time. For enhanced comparability, the number of accepted suggestions is then normed with the number of suggestions, so that this variable displays the success in the range [0;1]. The optimization progress represented by the Best Fitness (left y-axis) is visualized by a line graph on a logarithmic scale.

Citation

@misc{gabor2019benchmarking,
  title={Benchmarking Surrogate-Assisted Genetic Recommender Systems},
  author={Gabor et al. (2019)},
  year={2019},
  note={arXiv:1908.02880}
}

• arXiv: 1908.02880