Brainteaser Eval

brainteaser-eval

Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models — Han et al. (2025) (arXiv:2505.10844, 2025)

What this evaluates

Evaluates large language models' problem-solving capabilities using narrative-form brainteasers, probing their ability to generate correct final answers and employ creative, insight-based reasoning strategies rather than relying on brute-force or trial-and-error methods.

Datasets

• Braingle Math — total 242; splits: test (242); repo https://github.com/stephenxia1/brainteasers
• Braingle Logic — total 236; splits: test (236); repo https://github.com/stephenxia1/brainteasers

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy on the final answer. A prediction is counted as correct if it matches the ground truth answer provided in the dataset.
• creative_step_ratio — range: [0, 1]
  • Ratio of creative steps to total solution steps. Creative steps generate innovative insights or reduce problem complexity, while rudimentary steps involve straightforward computation or trial-and-error.

Input / output format

Input: Narrative-form brainteaser problem text (question) with optional metadata (difficulty rating, popularity score, hints).

Output: Final answer to the brainteaser, optionally accompanied by a step-by-step breakdown of the solution strategy.

Scoring recipe

def score(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    accuracy = correct / len(golds)
    return {"accuracy": accuracy}

Common pitfalls

• Problems are often informal, open-ended, or heuristic-based, meaning strict mathematical rigor is not always required; human intuition and pattern recognition are key.
• LLMs frequently default to brute-force or trial-and-error strategies even when insight-based solutions exist, which can skew strategy evaluation if only final answers are checked.
• Difficulty ratings are user-voted on a 1-4 scale and may not perfectly correlate with actual computational complexity or LLM performance.

Evidence (verbatim from paper)

The inclusion of these informal and heuristically-driven problems in Braingle Brainteaser allows us to assess LLMs not only on their ability to construct formal solutions to structured math and logic problems, but also on their ability to find creative patterns which align with human intuition. We observe that, by providing OpenAI o3 with detailed instructions and few-shot examples of step breakdown, it is able to restate the human solutions into correct, broken-down steps that arrive at the final answer based on manual inspection on 30 examples.

Citation

@misc{han2025brainteasers,
  title={Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models},
  author={Han et al. (2025)},
  year={2025},
  note={arXiv:2505.10844}
}

• arXiv: 2505.10844