qhjqhj00 sirius-robot-eval
Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment — Liu et al. (2022) (arXiv:2211.08416, 2022)
What this evaluates
Evaluates the policy success rate and human workload reduction of a human-in-the-loop robot learning framework over multiple deployment rounds on contact-rich manipulation tasks in simulation and real-world settings.
Datasets
- Sirius Robot Manipulation Tasks — total ?; splits: test (-1)
Metrics
success rate(primary) — range: percent- Percentage of trials where the robot successfully completes the task. In simulation, averaged over the top three performing checkpoints across 3 seeds, with 100 trials per checkpoint. In real-world, evaluated over 32 trials for the best checkpoint from a 5-checkpoint sweep.
human workload— range: percent- Percentage of intervention samples in the trajectories per round, used to track human workload reduction over deployment rounds.
Input / output format
Input: Robot state observations and task goals in task space; human interventions provided via a SpaceMouse during deployment.
Output: Action commands for the Franka Emika Panda robot arm and parallel jaw gripper.
Scoring recipe
def evaluate_policy(checkpoints, trials_per_cp=100, seeds=3):
all_rates = []
for seed in range(seeds):
for cp in checkpoints:
successes = sum(1 for _ in range(trials_per_cp) if run_trial(cp))
all_rates.append(successes / trials_per_cp)
final_success_rate = sum(sorted(all_rates, reverse=True)[:3]) / 3
workload = (num_intervention_samples / total_samples_per_round) * 100
return final_success_rate, workload
Common pitfalls
- Simulation success rate is not taken from the final checkpoint alone; it averages the top 3 performing checkpoints across 3 seeds to avoid outliers.
- Real-world evaluation uses only 1 seed due to high time cost, unlike the 3 seeds used in simulation.
- Human workload is strictly quantified as the percentage of intervention samples in trajectories, not subjective mental effort or time duration.
Evidence (verbatim from paper)
We benchmark human-in-the-loop deployment systems in two aspects: 1) Policy Performance. Our human-robot team achieves a reliable task success of 100%. Here we evaluate the success rate of the autonomous policy after each round of model update; and 2) Human Workload. We measure human workload as the percentage of intervention in the trajectories in each round. Simulation experiments: We evaluate the success rate of each method across 3 seeds. For each seed, we evaluate the success rate at a set of regularly spaced training checkpoints and record the average over the top three performing checkpoints to avoid outliers. For each checkpoint, we evaluate whether the agent successfully completed the task over 100 trials.
Citation
@misc{liu2022sirius,
title={Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment},
author={Liu et al. (2022)},
year={2022},
note={arXiv:2211.08416}
}
- arXiv: 2211.08416