Horizonbench Eval

horizonbench-eval

HorizonBench: Long-Horizon Personalization with Evolving Preferences — Li et al. (2026) (arXiv:2604.17283, 2026)

What this evaluates

Probes long-horizon personalization and belief-update capability. It tests whether models can track evolving user preferences across ~6 months of conversation history and correctly select responses aligned with updated preferences, rather than anchoring on outdated values.

Datasets

• HorizonBench — total 4245; splits: test (4245); repo https://github.com/stellalisy/HorizonBench

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of items where the model correctly selects the gold assistant response out of five options.
• pre-evolution distractor selection rate — range: [0, 1]
  • Proportion of wrong answers that select the option reflecting the user's original, outdated preference.

Input / output format

Input: Full conversation history (~6 months, ~4,300 turns, ~163K tokens) and five candidate assistant responses.

Output: Selection of exactly one of the five candidate responses.

Scoring recipe

correct = 0
distractor_wrong = 0
total_wrong = 0
for item in dataset:
    pred = model.generate(item.history, item.options)
    if pred == item.gold:
        correct += 1
    else:
        total_wrong += 1
        if pred == item.pre_evolution_distractor:
            distractor_wrong += 1
accuracy = correct / len(dataset)
distractor_rate = distractor_wrong / total_wrong if total_wrong > 0 else 0

Common pitfalls

• Truncating long histories by removing earlier turns can artificially lower performance or alter preference tracking, so results are reported on history-filtered items only.
• The 20% chance baseline applies to the 5-option task, but models often score below it due to systematic anchoring on the pre-evolution distractor rather than uniform random guessing.
• Position bias is minor (<1 pp difference from raw accuracy) and should not be cited as the primary explanation for performance gaps.

Evidence (verbatim from paper)

We report results on the combined multi-generator dataset (4,245 items from 360 users across three generators). Items exceeding a model’s context window are truncated by removing earlier conversation turns. Figure 3 reports per-model accuracy on the 4,245 history-filtered items.

Citation

@misc{li2026horizonbench,
  title={HorizonBench: Long-Horizon Personalization with Evolving Preferences},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.17283}
}

• arXiv: 2604.17283