Ubsoft Benchmark Eval

ubsoft-benchmark-eval

UBSoft: A Simulation Platform for Robotic Skill Learning in Unbounded Soft Environments — Lin et al. (2024) (arXiv:2411.12711, 2024)

What this evaluates

This evaluation protocol assesses the computational efficiency, physical realism, and policy-learning effectiveness of the UBSoft simulation platform for robotic tasks in unbounded soft environments. It benchmarks how well different algorithms (RL, trajectory optimization, heuristics) perform on eight manipulation and locomotion tasks, and evaluates the fidelity of open-loop sim-to-real transfer.

Datasets

• UBSoft Benchmark — total ?; splits: manipulation (4), locomotion (4)

Metrics

• reward (primary) — range: other
  • Sum of step-wise rewards over the episode horizon. Step rewards are task-specific: negative Chamfer distance for Sand Painting, height variation plus scooped particle count for Dig A Hole and Scoop Out, particle count above a threshold for Smooth Surface, and rigid body state metrics for locomotion tasks.
• Chamfer distance — range: other
  • Pairwise minimum distance between two point clouds (simulated particle sets), used to quantify simulation realism against a high-resolution MLS-MPM 256 reference.
• simulation speedup — range: other
  • Ratio of baseline simulator (MLS-MPM) runtime to UBSoft runtime for equivalent scene scales, tracking storage and time consumption.

Input / output format

Input: Discretized soft material state (particle positions/velocities on a spatial grid) and rigid body states (joint angles, velocities, poses) provided as observations per timestep.

Output: Robot control actions (joint torques/velocities for quadrupeds/humanoids, end-effector poses/trajectories for manipulators) executed over a fixed simulation horizon.

Scoring recipe

def evaluate_policy(predictions, gold, env):
    trajectory = env.step(predictions)
    reward = 0.0
    task = gold['task_name']
    if task == 'Sand Painting':
        reward = -chamfer_distance(trajectory.particles, gold['target_shape'])
    elif task in ['Dig A Hole', 'Scoop Out']:
        reward = height_variation(trajectory) + count_scooped(trajectory)
    elif task == 'Smooth Surface':
        reward = count_above_threshold(trajectory.particles, gold['threshold'])
    elif task in ['Quadruped Sand Walk', 'Quadruped Snow Walk', 'Humanoid Stand', 'Humanoid Walk']:
        reward = rigid_body_state_reward(trajectory, gold['target_pose'])
    return sum(reward)

Common pitfalls

• Soft materials have infinite degrees of freedom, so downsampled observations cannot fully capture subtle geometries, causing RL algorithms to struggle with manipulation tasks.
• Gradient-based optimization (DiffPhys) suffers from inaccurate gradients due to clipping during long-horizon backpropagation, making sampling-based methods (CMA-ES) more reliable for most tasks.
• Sim-to-real transfer requires precise physical parameter alignment (e.g., Young's modulus, friction angle); small mismatches break open-loop trajectory deployment.

Evidence (verbatim from paper)

We utilize a human-defined reward function as the direct evaluation metric because it most accurately reflects the training objectives. Rewards are designed based on Chamfer distance for Sand Painting, height variation and scooped particles for Dig A Hole and Scoop Out, and particles above a threshold for Smooth Surface. While for various locomotion challenges, the reward is derived directly from the state of the rigid bodies.

Citation

@misc{lin2024ubsoft,
  title={UBSoft: A Simulation Platform for Robotic Skill Learning in Unbounded Soft Environments},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2411.12711}
}

• arXiv: 2411.12711