Rsbench Eval

rsbench-eval

Language Model Evolutionary Algorithms for Recommender Systems: Benchmarks and Algorithm Comparisons — Liu et al. (2024) (arXiv:2411.10697, 2024)

What this evaluates

Evaluates the ability of LLM-based evolutionary algorithms to optimize session-based recommendation prompts across multiple objectives (accuracy, diversity, and fairness) simultaneously.

Datasets

• RSBench — total ?; splits: train (-1), val (-1)

Metrics

• HV (primary) — range: other
  • Hypervolume of the non-dominated set of recommendation prompts in the 3D objective space (accuracy, diversity, fairness) relative to a fixed reference point. Measures both convergence to the Pareto front and diversity/spread of solutions.

Input / output format

Input: A candidate set or population of recommendation prompts, optionally conditioned on task examples or constraints, provided to the LLM for evolutionary operations (initialization, crossover, mutation).

Output: Optimized recommendation prompts, typically formatted with and tags to delimit the final generated prompt.

Scoring recipe

def compute_hv(non_dominated_prompts, reference_point):
    # non_dominated_prompts: list of [accuracy, diversity, fairness] scores
    # reference_point: worst-case objective values defining the bounding box
    # Calculate the volume of the objective space dominated by the Pareto front
    hv = 0.0
    for point in non_dominated_prompts:
        hv += hypervolume_contribution(point, reference_point, non_dominated_prompts)
    return hv

Common pitfalls

• HV values are highly sensitive to the choice of the reference point; all algorithms must be evaluated against the exact same reference point for fair comparison.
• Raw execution time is heavily influenced by LLM API provider latency and network conditions, making token consumption a more reliable and standardized efficiency metric.
• Results are averaged over 5 independent runs with diverse random seeds to account for stochastic initialization and LLM sampling variance; single-run results are not comparable.

Evidence (verbatim from paper)

Table III presents the HV results obtained by these algorithms across multiple candidate sets generated with different random seeds. The results indicate that LLM-IBEA achieves the best overall performance, attaining the highest HV values on six RSBench problems (RSBench-1, RSBench-2, RSBench-5 to RSBench-7, and RSBench-9) with the training sets and on four problems (RSBench-1, RSBench-2, RSBench-5, and RSBench-6) with the validation sets.

Citation

@misc{liu2024lmea,
  title={Language Model Evolutionary Algorithms for Recommender Systems: Benchmarks and Algorithm Comparisons},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2411.10697}
}

• arXiv: 2411.10697