Interndata A1 Real Eval

interndata-a1-real-eval

InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy — Yang Tian et al. (2025) (arXiv:2511.16651, 2025)

What this evaluates

Evaluates the zero-shot sim-to-real transfer and generalization of a Vision-Language-Action (VLA) policy on diverse real-world and simulated manipulation tasks. It probes fundamental pick-and-place, articulated object manipulation, human-robot interaction, and long-horizon task composition capabilities.

Datasets

• InternData-A1 Real-World & Sim-to-Real Benchmarks — total ?; splits: test (-1)

Metrics

• average success rate (primary) — range: percent
  • Calculated as the number of successful rollouts divided by the total number of rollouts (30 per task). A trial is successful only if all task-specific physical and procedural criteria are met.

Input / output format

Input: Multi-view visual observations (head and wrist cameras) and natural language instructions describing the manipulation task.

Output: Robot action commands (joint positions/velocities or end-effector poses) for dual-arm manipulation.

Scoring recipe

successes = 0
for rollout in range(30):
    if check_task_success(task_id, rollout_state):
        successes += 1
return (successes / 30) * 100

Common pitfalls

• Success criteria are strictly defined per task (e.g., 'fully and correctly zipped'), so minor deviations count as failures.
• High stochasticity in real-world physics and perception requires averaging over 30 rollouts per task to obtain stable metrics.
• Sim-to-real transfer performance may vary significantly across different robot embodiments (Genie-1, ARX Lift-2, ARX AC One) due to hardware-specific control latencies.

Evidence (verbatim from paper)

For each task, we define 15 evaluation settings, and to reduce stochasticity, we run two trials per setting. In total, each task is evaluated with 30 rollouts, and we report the average success rate.

Citation

@misc{tian2025interndataa1,
  title={InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy},
  author={Yang Tian et al. (2025)},
  year={2025},
  note={arXiv:2511.16651}
}

• arXiv: 2511.16651