Pencil Puzzle Bench Eval

pencil-puzzle-bench-eval

Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning — Waugh (2026) (arXiv:2603.02119, 2026)

What this evaluates

Evaluates multi-step verifiable reasoning and agentic iteration on constraint-satisfaction puzzles. It probes a model's ability to plan, execute moves, check constraints step-by-step, and course-correct over long contexts.

Datasets

• Pencil Puzzle Bench — total 62231; splits: test (-1); HF approximatelabs/pencil-puzzle-bench

Metrics

• success rate (primary) — range: percent
  • Percentage of puzzles correctly solved by the model. Calculated as (number of solved puzzles / total puzzles evaluated) * 100.
• uplift — range: percent
  • Absolute percentage point gain from agentic iteration over direct-ask performance. Calculated as agentic success rate minus direct success rate.
• cost per success — range: other
  • Total recorded benchmark cost in USD divided by the number of successfully solved puzzles.

Input / output format

Input: Constraint-satisfaction pencil puzzles presented as initial board states with associated rules. Models receive the puzzle configuration and must generate moves or solutions.

Output: Iterative board state updates (moves) or a final solved board configuration. In agentic mode, models output moves, receive constraint-violation feedback, and continue until solved or max turns/time reached.

Scoring recipe

def compute_metrics(predictions, gold, outcomes):
    solved = sum(1 for o in outcomes if o == 'correct')
    total = len(predictions)
    success_rate = (solved / total) * 100
    return success_rate

Common pitfalls

• Infrastructure errors (timeouts, API failures) are not always filtered out, artificially lowering success rates and skewing cost analysis.
• Higher reasoning effort does not monotonically improve performance; some models perform worse at maximum effort due to reliability/capability tradeoffs.
• Cost per success is underreported when agentic runs terminate early, as token usage from completed turns before the error is not recorded.

Evidence (verbatim from paper)

Table 3: Top 15 models by best success rate. Direct ask on 300 puzzles; agentic on 30-puzzle baseline for most models, 60-puzzle expanded set for top 3.

Citation

@misc{waugh2026pencil,
  title={Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning},
  author={Waugh (2026)},
  year={2026},
  note={arXiv:2603.02119}
}

• arXiv: 2603.02119