# Sirius Robot Eval

> Evaluates the policy success rate and human workload reduction of a human-in-the-loop robot learning framework over multiple deployment rounds on contact-rich manipulation tasks in simulation and real-world settings. Use when the user wants to benchmark on Sirius Robot Manipulation Tasks, or asks about evaluating this task. Reports success rate.

- Skill: `qhjqhj00/sirius-robot-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/sirius-robot-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/sirius-robot-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/sirius-robot-eval

---


# sirius-robot-eval

> Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment — Liu et al. (2022) (arXiv:2211.08416, 2022)

## What this evaluates

Evaluates the policy success rate and human workload reduction of a human-in-the-loop robot learning framework over multiple deployment rounds on contact-rich manipulation tasks in simulation and real-world settings.

## Datasets

- **Sirius Robot Manipulation Tasks** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of trials where the robot successfully completes the task. In simulation, averaged over the top three performing checkpoints across 3 seeds, with 100 trials per checkpoint. In real-world, evaluated over 32 trials for the best checkpoint from a 5-checkpoint sweep.
- `human workload` — range: percent
  - Percentage of intervention samples in the trajectories per round, used to track human workload reduction over deployment rounds.

## Input / output format

**Input**: Robot state observations and task goals in task space; human interventions provided via a SpaceMouse during deployment.

**Output**: Action commands for the Franka Emika Panda robot arm and parallel jaw gripper.

## Scoring recipe

```python
def evaluate_policy(checkpoints, trials_per_cp=100, seeds=3):
    all_rates = []
    for seed in range(seeds):
        for cp in checkpoints:
            successes = sum(1 for _ in range(trials_per_cp) if run_trial(cp))
            all_rates.append(successes / trials_per_cp)
    final_success_rate = sum(sorted(all_rates, reverse=True)[:3]) / 3
    workload = (num_intervention_samples / total_samples_per_round) * 100
    return final_success_rate, workload
```

## Common pitfalls

- Simulation success rate is not taken from the final checkpoint alone; it averages the top 3 performing checkpoints across 3 seeds to avoid outliers.
- Real-world evaluation uses only 1 seed due to high time cost, unlike the 3 seeds used in simulation.
- Human workload is strictly quantified as the percentage of intervention samples in trajectories, not subjective mental effort or time duration.

## Evidence (verbatim from paper)

> We benchmark human-in-the-loop deployment systems in two aspects: 1) Policy Performance. Our human-robot team achieves a reliable task success of 100%. Here we evaluate the success rate of the autonomous policy after each round of model update; and 2) Human Workload. We measure human workload as the percentage of intervention in the trajectories in each round. Simulation experiments: We evaluate the success rate of each method across 3 seeds. For each seed, we evaluate the success rate at a set of regularly spaced training checkpoints and record the average over the top three performing checkpoints to avoid outliers. For each checkpoint, we evaluate whether the agent successfully completed the task over 100 trials.

## Citation

```bibtex
@misc{liu2022sirius,
  title={Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2211.08416}
}
```

- arXiv: 2211.08416