Robocoin Eval

robocoin-eval

RoboCOIN: An Open-Sourced Bimanual Robotic Data COllection for INtegrated Manipulation — Wu et al. (2025) (arXiv:2511.17441, 2025)

What this evaluates

Evaluates the effectiveness of a large-scale bimanual manipulation dataset and a hierarchical annotation framework on Vision-Language-Action models across different robotic platforms. It probes the models' ability to generalize across task complexities, leverage multi-resolution annotations, and benefit from trajectory quality filtering.

Datasets

• RoboCOIN — total 180000; splits: test (-1); repo https://github.com/FlagOpen/CoRobot

Metrics

• success_rate (primary) — range: percent
  • Calculated as the ratio of successfully completed trials to the total number of trials, reported as a percentage. Success is determined by task completion criteria defined per task (e.g., object placed correctly, drawer closed).
• trajectory_filtering_rate — range: percent
  • Percentage of trajectories removed by the RTML framework based on global, phase-wise, or metric-wise constraints.

Input / output format

Input: Multi-view camera images, natural language task instructions, and hierarchical annotations (trajectory-level scene context, segment-level subtask instructions, frame-level state descriptions) fed into VLA models.

Output: Continuous motor commands / action sequences for robotic arms and grippers.

Scoring recipe

def compute_success_rate(successful_trials, total_trials):
    return (successful_trials / total_trials) * 100.0

def compute_filtering_rate(removed_trajectories, total_trajectories):
    return (removed_trajectories / total_trajectories) * 100.0

Common pitfalls

• RTML filtering relies on empirically set thresholds that may not generalize across all scenarios or robot morphologies.
• Success rates vary significantly with task complexity and embodiment, so comparing results across different platforms or task grids requires careful normalization.
• The evaluation does not include mixed-embodiment training or cross-embodiment policy transfer, limiting claims about generalization.

Evidence (verbatim from paper)

As shown in Figure 8, the four configurations exhibited a clear progressive improvement in average success rates. GR00T-Coarse showed a modest 3% gain over the GR00T-Raw baseline, indicating that basic global filtering provides limited benefits. In contrast, GR00T-Fine achieved a more substantial 16% improvement, underscoring the significant impact of phase-level constraints on trajectory quality.

Citation

@misc{wu2025robocoin,
  title={RoboCOIN: An Open-Sourced Bimanual Robotic Data COllection for INtegrated Manipulation},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2511.17441}
}

• arXiv: 2511.17441