qhjqhj00 chess-puzzle-eval
Amortized Planning with Large-Scale Transformers: A Case Study on Chess — Ruoss et al. (2024) (arXiv:2402.04494, 2024)
What this evaluates
Evaluates a model's ability to play chess and solve tactical puzzles without explicit search, testing its capacity for long-horizon planning and generalization to novel board states.
Datasets
- Lichess puzzles — total 11000; splits: test (10000), val (1000); repo https://github.com/google-deepmind/searchless_chess
Metrics
Lichess Elo(primary) — range: other- Standard chess rating on the Lichess platform calculated from game outcomes against human and bot opponents.
Puzzle accuracy— range: [0, 1]- Percentage of puzzles solved correctly, requiring the full correct move sequence to be predicted greedily.
Action accuracy— range: [0, 1]- Percentage of top-1 predicted actions matching the gold action for a given board state.
Kendall's τ— range: [-1, 1]- Rank correlation coefficient measuring how well predicted action-values match the gold action-values.
Input / output format
Input: Chess board state (position) with the set of legal moves.
Output: Predicted action-values for all legal moves, or a single greedy move selection derived from those values.
Scoring recipe
def score_puzzle_accuracy(pred_sequences, gold_sequences):
correct = sum(1 for p, g in zip(pred_sequences, gold_sequences) if p == g)
return correct / len(gold_sequences)
def score_action_accuracy(pred_actions, gold_actions):
return sum(1 for p, g in zip(pred_actions, gold_actions) if p == g) / len(gold_actions)
def score_kendall_tau(pred_values, gold_values):
return kendalltau(pred_values, gold_values).correlation
Common pitfalls
- Models are evaluated without explicit search at test time; puzzle solving relies purely on greedy value estimation, not lookahead.
- Lichess Elo against bots differs from human Elo due to different player pools and bot exploitation strategies.
- Large models may overfit to training board states if the dataset size is insufficient, despite low training loss.
Evidence (verbatim from paper)
Table 1 shows the playing strength (internal tournament Elo, external Lichess Elo, and puzzle accuracy) of our large-scale transformers trained on the full (10M games) training set.
Citation
@misc{ruoss2024amortized,
title={Amortized Planning with Large-Scale Transformers: A Case Study on Chess},
author={Ruoss et al. (2024)},
year={2024},
note={arXiv:2402.04494}
}
- arXiv: 2402.04494