Cleanupbench Eval

cleanupbench-eval

CleanUpBench: Embodied Sweeping and Grasping Benchmark — Li et al. (2025) (arXiv:2508.05543, 2025)

What this evaluates

Evaluates embodied cleaning agents in physics-accurate indoor simulations, probing their ability to perform sweeping and grasping tasks across diverse cluttered scenes. It measures task completion, spatial coverage efficiency, motion quality, and collision safety under strict time limits.

Datasets

• CleanUpBench — total 20; splits: test (20)

Metrics

• TCR (primary) — range: [0, 1]
  • Overall Task Completion Rate, calculated as the fraction of successfully completed cleaning tasks (sweeping and grasping) out of total targets.
• TCR_S — range: [0, 1]
  • Sweep Task Completion Rate, measuring the fraction of sweeping targets successfully cleared.
• TCR_G — range: [0, 1]
  • Grasp Task Completion Rate, measuring the fraction of grasping targets successfully picked up.
• ME — range: other
  • Motion Efficiency, defined as total travel distance in meters divided by the number of targets.
• SR — range: [0, 1]
  • Sweep Redundancy, quantifying overlapping or repeated sweeping actions relative to optimal coverage.
• CR — range: [0, 1]
  • Coverage Rate, measuring the proportion of the floor area successfully swept.
• FT — range: other
  • Task Completion Time, the total elapsed seconds until task finish or timeout.
• CT — range: other
  • Computation Time, the processing time in seconds required for decision-making per step.
• Vel_avg — range: other
  • Average Velocity, the mean speed in meters per second during operation.
• Col — range: other
  • Total Collision Count, the number of physical collisions with obstacles or objects.

Input / output format

Input: A physics-accurate simulation of an indoor cluttered environment with a mobile robot/manipulator, sensor modalities, and task goals (sweeping, grasping, or dual-mode) within a 300-second time limit.

Output: Continuous or discrete action commands for robot locomotion and arm manipulation, executed step-by-step until task completion or the 300-second timeout.

Scoring recipe

def score(trajectory, gold_tasks, time_limit=300):
    total_targets = len(gold_tasks)
    swept = count_completed_sweeps(trajectory)
    grasped = count_completed_grasps(trajectory)
    tcr = (swept + grasped) / total_targets
    tcr_s = swept / count_sweep_targets(gold_tasks)
    tcr_g = grasped / count_grasp_targets(gold_tasks)
    me = trajectory.total_distance / total_targets
    sr = 1.0 - (trajectory.swept_area / total_area)
    cr = trajectory.swept_area / total_area
    ft = min(trajectory.end_time, time_limit)
    ct = trajectory.computation_time
    vel_avg = trajectory.total_distance / ft if ft > 0 else 0
    col = trajectory.collision_count
    return {'TCR': tcr, 'TCR_S': tcr_s, 'TCR_G': tcr_g, 'ME': me, 'SR': sr, 'CR': cr, 'FT': ft, 'CT': ct, 'Vel_avg': vel_avg, 'Col': col}

Common pitfalls

• Evaluating sweep-only or grasp-only baselines on both tasks without using the decomposed TCR_S/TCR_G metrics, which unfairly penalizes them with zero scores.
• Ignoring the strict 300-second time limit, which truncates trajectories and artificially deflates completion rates and inflates time-based metrics.
• Directly comparing single-robot and multi-robot methods on raw TCR without accounting for coordination overhead, computational time (CT), and collision penalties (Col).

Evidence (verbatim from paper)

TCR: Overall Task Completion Rate. TCR $_S$ : Sweep Task Completion Rate. TCR $G$ : Grasp Task Completion Rate. ME: Motion Efficiency (m/target). SR: Sweep Redundancy. CR: Coverage Rate. FT: Task Completion Time (s). CT: Computation Time (s). Vel ${\text{avg}}$ : Average Velocity (m/s). Col: Total Collision Count.

Citation

@misc{li2025cleanupbench,
  title={CleanUpBench: Embodied Sweeping and Grasping Benchmark},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2508.05543}
}

• arXiv: 2508.05543