qhjqhj00 domino-eval
Towards Generalizable Robotic Manipulation in Dynamic Environments — Fang et al. (2026) (arXiv:2603.15620, 2026)
What this evaluates
Evaluates a robot policy's ability to perform manipulation tasks in environments with moving objects and dynamic spatiotemporal changes. It probes the model's capacity for historical context integration and future state anticipation to maintain control stability and task success under motion.
Datasets
- DOMINO@0.1 — total 110000; splits: train (-1), test (-1); repo https://github.com/H-EmbodVis/DOMINO
Metrics
Success Rate (SR)(primary) — range: percent- Percentage of evaluation episodes where the robot successfully completes the specified manipulation task, calculated as (successful_episodes / total_episodes) * 100.
Manipulation Score (MS)— range: other- Continuous metric quantifying the quality of interaction with moving targets, reflecting control stability, trajectory tracking accuracy, and temporal consistency during manipulation.
Input / output format
Input: Multi-view RGB images, historical optical flow frames, object-centric predictive queries, robot proprioceptive state, and natural language task instructions.
Output: Robot action commands (e.g., joint velocities or end-effector poses) for each control step.
Scoring recipe
def compute_sr(predictions, gold):
success_count = 0
for pred, gold in zip(predictions, gold):
if check_task_completion(pred, gold):
success_count += 1
return (success_count / len(gold)) * 100
def compute_ms(predictions, gold):
scores = [calculate_interaction_quality(pred, gold) for pred, gold in zip(predictions, gold)]
return sum(scores) / len(scores)
Common pitfalls
- Assuming fine-tuning on dynamic data alone bridges the static-to-dynamic performance gap (improvements are <3%).
- Using single-frame observations instead of historical context causes severe degradation in dynamic settings.
- Confusing zero-shot static-to-dynamic transfer (S→D) with fine-tuned dynamic-to-dynamic (D→D) evaluation protocols.
Evidence (verbatim from paper)
We primarily evaluate on our proposed DOMINO@0.1 benchmark, reporting the Success Rate (SR) and Manipulation Score (MS). ... Specifically, PUMA achieves the highest average success rate of 17.20%, substantially outperforming recent strong baselines ... Furthermore, our method attains a peak Manipulation Score of 34.97, indicating a higher quality of interaction with moving targets.
Citation
@misc{fang2026domino,
title={Towards Generalizable Robotic Manipulation in Dynamic Environments},
author={Fang et al. (2026)},
year={2026},
note={arXiv:2603.15620}
}
- arXiv: 2603.15620