Game Of 24 Eval

game-of-24-eval

Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models — Sel et al. (2023) (arXiv:2308.10379, 2023)

What this evaluates

Evaluates an LLM's ability to perform algorithmic search and recursive reasoning within a single generation window, without external tree search or iterative prompting. It probes systematic exploration, pruning, and backtracking capabilities in a mathematical constraint satisfaction task.

Datasets

• Game of 24 — total 100; splits: test (100)

Metrics

• Success rate (primary) — range: percent
  • Percentage of instances where the model generates a valid mathematical expression using exactly the four input numbers and allowed operations (+, -, *, /) that evaluates to 24. Calculated as (correct predictions / total instances) * 100.
• Avg. Queries — range: other
  • Average number of LLM generation calls or API requests required per instance to produce a final answer.

Input / output format

Input: Four integers representing the card values. Prompts include a 5-shot in-context setup with DFS-style search trajectories as examples.

Output: A single mathematical expression string using the four numbers and +, -, *, / operators that equals 24.

Scoring recipe

def score_game_of_24(predictions, gold_numbers):
    correct = 0
    for pred in predictions:
        try:
            # Extract numbers from prediction and verify they match gold
            pred_nums = extract_numbers(pred)
            if set(pred_nums) != set(gold_numbers):
                continue
            # Evaluate expression safely
            if evaluate_expression(pred) == 24:
                correct += 1
        except Exception:
            continue
    return (correct / len(gold_numbers)) * 100

Common pitfalls

• Token limits frequently cause 'out-of-token' errors before a solution is found, artificially deflating success rates.
• LLMs may internally discover the correct solution but fail to output it ('non-finalization error'), requiring manual resolution to isolate true algorithmic capability.
• Baselines like ToT rely on external memory and backtracking, making direct success-rate comparisons to single-query generation methods misleading without accounting for query overhead.

Evidence (verbatim from paper)

Table 1: Game of 24: success rates and the average number of LLM queries for each example. For an attempt to be considered successful, it must derive a total of 24 using the exact numbers provided and only the allowed operations.

Citation

@misc{sel2023algorithmofthoughts,
  title={Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models},
  author={Sel et al. (2023)},
  year={2023},
  note={arXiv:2308.10379}
}

• arXiv: 2308.10379