Herobench Eval

herobench-eval

HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds — Anokhin et al. (2025) (arXiv:2508.12782, 2025)

What this evaluates

Evaluates long-horizon planning and structured reasoning in a grid-based RPG virtual environment. Agents must generate multi-step plans involving resource gathering, crafting, and combat, requiring integration of numerical calculations with action sequencing.

Datasets

• HeroBench — total ?; splits: base (-1), harder (-1); repo https://github.com/stefanrer/HeroBench

Metrics

• Success % (primary) — range: percent
  • Percentage of tasks where the model's generated plan and executed actions fully satisfy the task requirements without high-level planning or low-level execution errors.
• Score — range: percent
  • Progress score reflecting task completion quality and efficiency, reported as mean ± standard deviation across difficulty levels.
• pass@k — range: percent
  • Success rate calculated over k independent attempts per task, measuring the probability of solving a task within k trials.

Input / output format

Input: Grid-based RPG environment state, task instructions specifying objectives (e.g., resource gathering, crafting, combat), and environmental constraints or distractor items.

Output: High-level gear selection plan, low-level action sequence, and valid Python code to execute the plan within the environment.

Scoring recipe

def score_hero_bench(predictions, gold_tasks):
    success_count = 0
    total_score = 0
    for pred, task in zip(predictions, gold_tasks):
        errors = analyze_errors(pred)
        if errors['invalid_code'] == 0 and errors['high_level'] == 0 and errors['execution'] == 0:
            success_count += 1
            total_score += pred.progress_score
    return {
        'Success %': (success_count / len(gold_tasks)) * 100,
        'Score': total_score / len(gold_tasks)
    }

Common pitfalls

• Models often fail at high-level planning (gear selection) rather than low-level execution, so error analysis must separate these two failure modes.
• Invalid Python code generation is counted as a distinct failure type and must be parsed before attempting environment execution.
• Adversarial distractor items in noise-augmented tasks require robust filtering, as models frequently select suboptimal gear due to irrelevant environmental information.

Evidence (verbatim from paper)

Table 1: Mean performance of all evaluated models across nine base task difficulty levels in HeroBench. Columns show success rate (%), score (mean ± SD), and tokens (mean ± SD).

Citation

@misc{anokhin2025herobench,
  title={HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds},
  author={Anokhin et al. (2025)},
  year={2025},
  note={arXiv:2508.12782}
}

• arXiv: 2508.12782