qhjqhj00 maniskill2-eval
A Two-stage Fine-tuning Strategy for Generalizable Manipulation Skill of Embodied AI — Fang Gao et al. (2023) (arXiv:2307.11343, 2023)
What this evaluates
Evaluates the generalization and robustness of embodied AI manipulation policies across soft-body, rigid-body, and assembly tasks in a simulated environment.
Datasets
- ManiSkill2 — total ?; splits: test (-1)
Metrics
success rate(primary) — range: [0, 1]- The proportion of simulation episodes in which the robot successfully completes the specified manipulation task.
Input / output format
Input: Visual observations (RGB/depth) and proprioceptive states from the ManiSkill2 simulation environment.
Output: Continuous action commands (e.g., joint velocities or end-effector poses) for the robotic manipulator.
Scoring recipe
def compute_success_rate(rollouts):
total = len(rollouts)
successes = sum(1 for r in rollouts if r['success'] == 1)
return successes / total
Common pitfalls
- Overfitting to specific task dynamics during the first-stage fine-tuning.
- Sensitivity to hyperparameter scaling factors for batch size and sample density during the second stage.
Evidence (verbatim from paper)
The results of these experiments, as shown in Table I, indicate that the scale of 0.9 for the batch size and 0.875 for the number of samples yielded better performance. To further illustrate the effectiveness of our two-stage fine-tuning strategy, we visualized the trend line of the success rate with and without our strategy in Fig. 2.
Citation
@misc{gao2023twostage,
title={A Two-stage Fine-tuning Strategy for Generalizable Manipulation Skill of Embodied AI},
author={Fang Gao et al. (2023)},
year={2023},
note={arXiv:2307.11343}
}
- arXiv: 2307.11343