nvlabs ASPIRE Experiment Pipeline
What Is ASPIRE
ASPIRE — LLMs write Python code to control a robot via a structured API. Code runs in MuJoCo (Robosuite).
7 Robosuite tasks:
| Task | Type | Camera key |
|---|---|---|
cube_lifting |
Single-arm | robot0_robotview |
cube_restack |
Single-arm | robot0_robotview |
cube_stack |
Single-arm | robot0_robotview |
nut_assembly |
Single-arm | robot0_robotview |
spill_wipe |
Single-arm | robot0_robotview |
two_arm_lift |
Bimanual | robot0_robotview |
two_arm_handover |
Bimanual | robot0_robotview |
Pipeline Modes
| Mode | File | When |
|---|---|---|
| Robosuite Baseline | run-baseline.md | Collect baseline on all 7 tasks |
| Robosuite Fix Loop | main-agent-prompt.md | Coordinator guide: dispatch one subagent per GPU to debug failures |
| Robosuite Fix Loop Subagent | subagent-prompt.md | Self-contained prompt template for dispatching one task to a background subagent |
Reference Files
| File | Covers |
|---|---|
| run-baseline.md | Launch command, config paths, output structure for all 7 tasks |
| api-reference.md | Full API functions, output structure, TraceLogger format, source files |
| clean-task-slate.md | Checklist for resetting a task to clean slate before rerunning fix loop |
Companion Skills
| Skill | Covers |
|---|---|
grasp |
Pick-and-place code template, pre-grasp/lower/close/lift/place skeleton |
localize |
Perception server (SAM3/Molmo) prompting strategy, per-object prompt registry |
transport |
Motion patterns for moving objects between locations — multi-step waypoints, safe transit sequences, interpolated Cartesian moves, collision avoidance during transport |
Setup
Two venvs:
.venv-robosuite(Python 3.10) — Robosuite replay/eval:replay_trial_robosuite.py.venv-liberoor.venv-perception— perception servers only (handled bystart_perception_servers.sh)
Always use .venv-robosuite/bin/python3 for Robosuite replay/eval. Never use system python after setup.
Perception Servers (required before any experiment)
tmux new -s aspire-perception
cd "$ASPIRE_ROOT"
ASPIRE_PERCEPTION_PYTHON=.venv-libero/bin/python3 \
bash scripts/common/start_perception_servers.sh --with-molmo
for p in 8114 8115 8116 8122; do
echo "port $p: $(curl -s -o /dev/null -w '%{http_code}' --max-time 3 http://127.0.0.1:$p/health)"
done
# 404/non-000 = UP for 8114-8116, 000 = DOWN. Molmo health is /v1/models on 8122.
SAM3 uses gated Hugging Face weights; authenticate before startup. GraspNet
requires the pinned Contact-GraspNet submodule and the perception environment
to include --extra contactgraspnet; the startup script verifies and applies
the compatibility patch. Molmo starts by default; --with-molmo aborts unless
the perception environment provides a vllm executable, so pass --no-molmo
to skip it and give up point-prompt fallback. Keep
servers in tmux or another persistent terminal; one-off background shells can
exit and take child server processes down with them.
| Server | Port | GPU | Required for |
|---|---|---|---|
| SAM3 | 8114 | Configured GPU | All runs |
| GraspNet | 8115 | Configured GPU | All runs |
| PyRoKi | 8116 | CPU | All runs |
| Molmo | 8122 | Configured GPU | Point-prompt fallback |
GPU layout: Derive server and worker assignments from the active host configuration.
Running Experiments
Scripts use tyro.cli with a named args parameter — require the --args. prefix.
The examples assume SIM_GPU, SIM_GPUS, and DEBUG_TRIAL_ID are set from
the active host configuration and development partition.
# Replay fix code on a single trial
MUJOCO_GL=egl CUDA_VISIBLE_DEVICES="$SIM_GPU" TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 \
.venv-robosuite/bin/python3 scripts/robosuite/replay_trial_robosuite.py \
--args.config env_configs/robosuite/cube_lifting_multimodel_aspire_traced.yaml \
--args.trial "$DEBUG_TRIAL_ID" \
--args.replay-code /tmp/fix_attempt.py \
--args.output-dir ./outputs/debug_fix
# Interactive REPL (all API functions in scope)
MUJOCO_GL=egl CUDA_VISIBLE_DEVICES="$SIM_GPU" TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 \
.venv-robosuite/bin/python3 scripts/robosuite/replay_trial_robosuite.py \
--args.config env_configs/robosuite/cube_lifting_multimodel_aspire_traced.yaml \
--args.trial "$DEBUG_TRIAL_ID" \
--args.interactive \
--args.output-dir /tmp/repl_out
Config path pattern: env_configs/robosuite/<task>_multimodel_aspire_traced.yaml — all 7 tasks (including two_arm_handover) follow this single scheme. See the task reference table in main-agent-prompt.md for exact paths.
Trial split: Read the development and held-out partitions from the active experiment config; keep them disjoint and lock held-out trials during debugging.
Critical Rules
- NEVER git push — local commits only
- No forbidden APIs:
sim.data.body_xpos,sim.data.get_site_xpos,sim.data.set_joint_qpos,sim.model.*,sim.data.qpos,sim.forward(),env._step_once() replay_trial_robosuite.pyuses--args.prefix (tyro wraps theargsparameter)MUJOCO_GL=egl+TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1for all runs- Log to
docs/logs/YYYY-MM-DD.mdafter significant work - Update the experiment's
skills/(.claude/robosuite/fix-loop/skills/) after any new pattern discovered
Monitoring
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
for gpu in $SIM_GPUS; do
procs=$(nvidia-smi -i $gpu --query-compute-apps=pid --format=csv,noheader,nounits 2>/dev/null | grep -c '[0-9]')
if [ "$procs" -eq 0 ]; then echo "GPU $gpu: FREE"; else echo "GPU $gpu: BUSY ($procs processes)"; fi
done