Dre Bench Eval

dre-bench-eval

Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation — Yang et al. (2025) (arXiv:2506.02648, 2025)

What this evaluates

Evaluates large language models' fluid intelligence and abstract rule generalization across four hierarchical cognitive levels (Attribute, Spatial, Sequential, Conceptual). It probes the model's ability to dynamically adapt to varying task complexity and apply learned rules to novel grid-based reasoning problems.

Datasets

• DRE-Bench — total ?; splits: test (-1); repo https://github.com/yangyue5114/DRE-Bench

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy: the proportion of samples where the model's predicted output grid exactly matches the ground-truth output grid.

Input / output format

Input: Text-based grid representations of input-output pairs and a test input, optionally accompanied by auxiliary visual formats (single-image or multi-image) in ablation studies.

Output: A grid representation that must exactly match the ground-truth output grid.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)

Common pitfalls

• Assuming auxiliary visual information (single/multi-image) consistently improves performance; ablation shows it often degrades or has no effect on text-only baselines.
• Expecting inference time scaling to compensate for high-level reasoning deficits; longer latency does not improve performance on complex Level-3/4 tasks.
• Overestimating model capabilities at Level 4 (Conceptual); all evaluated models fail to solve these tasks even in simplest cases.

Evidence (verbatim from paper)

In the DRE-Bench benchmark, accuracy serves as the primary evaluation metric, defined as the proportion of samples for which the model’s output grid exactly matches the ground-truth output grid.

Citation

@misc{yang2025drebench,
  title={Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2506.02648}
}

• arXiv: 2506.02648