Bench Push Eval

bench-push-eval

Bench-Push: Benchmarking Pushing-based Navigation and Manipulation Tasks for Mobile Robots — Zhong et al. (2025) (arXiv:2512.11736, 2025)

What this evaluates

Evaluates the transferability and performance of reinforcement learning policies for mobile robot navigation and pushing-based manipulation tasks. It probes how well policies trained in simulation handle real-world sim-to-real gaps, clutter, and sparse rewards across varying obstacle densities.

Datasets

• Bench-Push (Maze & Box-Delivery) — total ?; splits: test (-1)

Metrics

• $S_{\text{manip}}$ (primary) — range: [0, 1]
  • Task success rate: fraction of episodes where the robot successfully pushes the box to the receptacle. Values are normalized scores where higher is better.
• $E_{\text{nav}}$ — range: [0, 1]
  • Navigation efficiency: normalized score measuring path/collision efficiency during maze traversal. Higher is better.
• $I_{\text{nav}}$ — range: [0, 1]
  • Navigation interaction effort: normalized score measuring physical interaction intensity during navigation. Higher is better.

Input / output format

Input: Overhead camera observations tracking robot pose and 3D-printed box positions at 10Hz, provided as state vectors to the deployed policy.

Output: Continuous control commands (velocity/steering) for the TurtleBot3 Burger.

Scoring recipe

def evaluate(episodes, task_type):
    success = 0
    eff_sum, int_sum = 0.0, 0.0
    for ep in episodes:
        if task_success(ep): success += 1
        eff_sum += normalize_efficiency(ep)
        int_sum += normalize_interaction(ep)
    n = len(episodes)
    return {
        'S_manip': success / n,
        'E_manip': eff_sum / n,
        'I_manip': int_sum / n
    }

Common pitfalls

• Physical testbed scores are consistently lower than simulation due to sim-to-real gaps (localization noise, control imperfections, box slipping), which should not be mistaken for policy failure.
• Metrics degrade monotonically with increased obstacle/box counts; comparing across different clutter levels without normalization is misleading.
• Interaction effort and efficiency are normalized scores, not raw physical units, so cross-environment absolute comparisons require caution.

Evidence (verbatim from paper)

TABLE IV: Results for Box-Delivery in the physical testbed and Bench-Push simulations. $\uparrow$ indicates higher is better. ... Smanip↑ Emanip↑ Imanip↑

Citation

@misc{zhong2025benchpush,
  title={Bench-Push: Benchmarking Pushing-based Navigation and Manipulation Tasks for Mobile Robots},
  author={Zhong et al. (2025)},
  year={2025},
  note={arXiv:2512.11736}
}

• arXiv: 2512.11736