Park Cleaning Benchmark

park-cleaning-benchmark

Cleaning Robots in Public Spaces: A Survey and Proposal for Benchmarking Based on Stakeholders Interviews — Memmesheimer et al. (2024) (arXiv:2407.16393, 2024)

What this evaluates

Evaluates autonomous cleaning robots' ability to navigate public park pathways, perceive and collect diverse litter types, avoid obstacles, and operate within strict physical and safety constraints.

Datasets

• Park Cleaning Benchmark — total ?; splits: test (-1)

Metrics

• collected_items_weight_or_count (primary) — range: kg or count or seconds
  • Total weight (kg) or count of successfully collected items from the pathway and wayside. Alternatively, total time (seconds) to complete the cleaning task.

Input / output format

Input: Real-world park environment containing a pathway, adjacent wayside (up to 1m from path), various litter types (glass, plastic cups, paper, cigarette butts) with varying size/material/moisture, and static/dynamic obstacles (curbs, cycle barriers, benches, parked cars).

Output: Autonomous navigation and manipulation actions to collect litter while avoiding obstacles and adhering to platform constraints (≤150kg, ≤1.2×0.75×1m, emergency button, obstacle avoidance).

Scoring recipe

def score(collected_items, task_time):
    # Primary metric: weight or count of collected items
    score = sum(item.weight for item in collected_items)
    # Bonus for hard-to-reach litter
    for item in collected_items:
        if item.location == 'hard_to_reach':
            score += bonus_points
    return score

Common pitfalls

• Proposed as a conceptual scenario without released code or fixed dataset splits
• Scoring metric is ambiguous: paper allows either collected item weight/count OR completion time
• Technical acceptance (size/weight limits, emergency button, obstacle avoidance) must pass before benchmark scoring begins

Evidence (verbatim from paper)

The benchmark scoring considers the amount of collected items (in weight or amount) or the time required to achieve the cleaning task. ... The technical requirements are defined as follows: Weight ≤ 150kg, Size ≤ 1.2m×0.75m×1m (L×W×H), Well visible and reachable emergency button, Obstacle avoidance (avoiding people, animals, pathway users and other obstacles)

Citation

@misc{memmesheimer2024cleaningrobots,
  title={Cleaning Robots in Public Spaces: A Survey and Proposal for Benchmarking Based on Stakeholders Interviews},
  author={Memmesheimer et al. (2024)},
  year={2024},
  note={arXiv:2407.16393}
}

• arXiv: 2407.16393