Handful Bench Eval

handful-bench-eval

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness — Foong et al. (2026) (arXiv:2604.25126, 2026)

What this evaluates

Evaluates sequential dexterous manipulation by requiring a robot to first grasp a target object and then perform a specific downstream task (e.g., pushing, pressing, twisting, pulling, or picking a second object) while maintaining the grasp. It probes the policy's ability to allocate finger resources and maintain stable contacts to satisfy competing subtask constraints.

Datasets

• HANDFUL-Bench — total ?; splits: test (-1)

Metrics

• terminal success rate ($p_{st}$) (primary) — range: [0, 1]
  • Binary metric indicating whether the task is successfully completed at the final timestep of the episode. Episodes run for the full duration rather than terminating upon first success.

Input / output format

Input: Robot state observations including arm and hand joint positions/velocities, palm and fingertip poses, wrist position, subtask-specific observations (object poses, relative poses, object half sizes, articulation joint positions), and a one-hot vector of active finger indices.

Output: 23-dimensional action vector: 16 delta joint commands for the hand and 7 for the arm, processed via a PD joint delta position controller.

Scoring recipe

def compute_metric(trajectory, task_config):
    state = init_state(task_config)
    for action in trajectory:
        state = step_simulation(state, action)
    return 1.0 if check_final_success(state, task_config) else 0.0

Common pitfalls

• Policies may terminate early upon first success, but the benchmark requires running episodes for the full duration to evaluate stability throughout the subtask.
• Initial grasps optimized only for force closure may fail downstream tasks because they occupy fingers or hand spaces needed for subsequent actions like pushing or twisting.

Evidence (verbatim from paper)

In HANDFUL-Bench, while the first grasping subtask is shared across all tasks, the second subtasks impose distinct and often competing demands on finger allocation, contact patterns, and space within the hand. ... Finally, to encourage stable grasps throughout, episodes run for the full duration of each subtask rather than terminating upon first success. We thus use terminal success rate $p_{st}$, whether the task is successfully completed at the final timestep, as our primary evaluation metric.

Citation

@misc{foong2026handful,
  title={HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness},
  author={Foong et al. (2026)},
  year={2026},
  note={arXiv:2604.25126}
}

• arXiv: 2604.25126