Peek Robot Zero Shot Eval

peek-robot-zero-shot-eval

PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies — Jesse Zhang et al. (2025) (arXiv:2509.18282, 2025)

What this evaluates

Evaluates zero-shot generalization and visual/semantic robustness of robot manipulation policies when transferred to new real-world setups, visual clutter, and unseen object configurations.

Datasets

• Franka Sim-to-Real Custom Setup — total 2500; splits: test (-1)
• BRIDGE-v2 — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Proportion of successful task completions out of total evaluation trials. Calculated as successes / total_trials per task variation.

Input / output format

Input: RGB images, depth/point clouds (for 3D policies), and language task instructions. PEEK overlays predicted end-effector paths and task-relevant masking points onto the observations.

Output: Robot end-effector actions/poses for manipulation tasks.

Scoring recipe

successes = 0
total = 0
for task_var in [Basic, Clutter, Semantic]:
    for trial in range(5):
        if task_completion_is_successful():
            successes += 1
        total += 1
return successes / total

Common pitfalls

• Confusing simulation data collection (2.5k trajectories) with the actual zero-shot evaluation, which occurs entirely in the real world or on a different WidowX setup.
• Assuming the metric is explicitly defined in the text; it relies on standard robotic task success rates without a formal mathematical formula provided.
• Overlooking that PEEK is policy-agnostic and evaluated across multiple base architectures (3DDA, ACT, π₀), not just a single model.

Evidence (verbatim from paper)

Each policy is evaluated for 5 trials per task, totaling 220 evaluations across 4 methods and 11 variations. We perform 5 evals per task with randomized object locations.

Citation

@misc{zhang2025peek,
  title={PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies},
  author={Jesse Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.18282}
}

• arXiv: 2509.18282