Builderbench Eval

builderbench-eval

BuilderBench -- A benchmark for generalist agents — Ghugare et al. (2025) (arXiv:2510.06288, 2025)

What this evaluates

Evaluates an agent's ability to learn embodied reasoning, long-horizon planning, and physical/geometric intuition through self-supervised exploration, and generalizes these skills to construct unseen block structures.

Datasets

• BuilderBench — total 42; splits: train (-1), test (42); repo https://github.com/RajGhugare19/builderbench

Metrics

• success_rate (primary) — range: [0, 1]
  • Binary success metric indicating whether the agent's final environment state matches the target block structure goal within a defined tolerance. Calculated as the fraction of test tasks completed successfully.

Input / output format

Input: State observations of the MuJoCo simulation (robotic hand pose, block positions) and a target block structure goal.

Output: Control actions for the robotic hand (locomotion, grasping, throwing, pick-and-place) within the MuJoCo environment.

Scoring recipe

def compute_metric(predictions, gold):
    successes = 0
    for goal in gold:
        final_state = predictions[goal]
        if is_structurally_equivalent(final_state, goal, tol=0.1):
            successes += 1
    return successes / len(gold)

Common pitfalls

• Agents may memorize specific block arrangements instead of learning general construction principles.
• The 'debug' mode allows direct access to test-time goals during training, which inflates performance compared to the strict self-supervised exploration protocol.
• Success tolerance thresholds for matching the target structure are not explicitly defined, requiring careful implementation of geometric/physical matching logic.

Evidence (verbatim from paper)

During training, agents must discover such skills through practice. During testing, agents must use those skills to build unseen structures... We open-source BuilderBench, a task-suite of over 40 tasks to evaluate the performance of agents. Each task corresponds to a different block structure.

Citation

@misc{ghugare2025builderbench,
  title={BuilderBench -- A benchmark for generalist agents},
  author={Ghugare et al. (2025)},
  year={2025},
  note={arXiv:2510.06288}
}

• arXiv: 2510.06288