Remote Training

Remote Training Infrastructure

This skill manages the Positronic training infrastructure on Nebius GPU VMs. It covers Docker image management, VM lifecycle, training jobs, dataset generation, and inference server deployment.

S3 Convention

s3://interim/{dataset}/{vendor}/{codec}/          — converted LeRobot datasets
s3://checkpoints/{dataset}/{vendor}/{codec_or_experiment}/  — training output
s3://inference/{dataset}/{date_or_exp}/{vendor}/  — inference eval results

Every run writes run_metadata_*.yaml with the full CLI command — read it to reconstruct the pipeline.

Current Artifacts

sim_stack (automated testing — @positronic.cfg.ds.phail.sim_stack_cubes):

phail_unified (production — @positronic.cfg.ds.phail.phail_unified):

Two LeRobot Versions

Positronic ships two LeRobot integrations because other vendors (GR00T, OpenPI) depend on the 0.3.3 dataset format:

lerobot-0_3_3-convert is also used for GR00T and OpenPI dataset conversion.

Machines

Only GR00T and OpenPI training/inference require H100. Everything else runs on desktop.

VM Management

IMPORTANT: Always check if a VM is running a job before using it.

# Check connectivity and running containers
ssh -o ConnectTimeout=5 vertix@vm-train 'echo connected' 2>&1
docker --context vm-train ps 2>/dev/null

# Start a stopped VM
../internal/scripts/start.sh train   # or train2, train3

Docker Images

Images are tagged by branch name. make push in docker/ auto-derives the tag from the current git branch.

cd /home/vertix/dev/positronic/docker
make push-training  # Just positro/positronic
make push-groot     # positro/gr00t (rebuild base first if ../gr00t changed)
make push-openpi    # positro/openpi (rebuild base first if ../openpi changed)
make push           # All images

For cross-repo base image rebuilds: cd ../gr00t/docker && make push then cd ../positronic/docker && make push-groot.

Pipeline

All commands run from docker/ directory. Use CACHE_ROOT=/home/vertix when targeting remote Docker contexts from Mac.

1. Convert Dataset

# GR00T / OpenPI / LeRobot 0.3.3 — use lerobot-0_3_3-convert
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always lerobot-0_3_3-convert convert \
  --dataset.dataset=@positronic.cfg.ds.phail.sim_stack_cubes \
  --dataset.codec=@positronic.vendors.gr00t.codecs.ee_rot6d \
  --output_dir=s3://interim/sim_stack/groot/ee_rot6d/

# LeRobot 0.4.x (SmolVLA) — use lerobot-convert
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always lerobot-convert convert \
  --dataset.dataset=@positronic.cfg.ds.phail.sim_stack_cubes \
  --dataset.codec=@positronic.vendors.lerobot.codecs.ee \
  --output_dir=s3://interim/sim_stack/lerobot_04/ee/

Default codecs: groot ee_rot6d, lerobot ee, openpi ee.

2. Train

Each vendor has a training script at positronic/vendors/{vendor}/train.py with usage examples in its docstring. Read the script's docstring for available subcommands and parameters.

General pattern:

# From docker/ directory
[CACHE_ROOT=/home/vertix] docker [--context <machine>] compose run --rm --pull always <service> \
  [subcommand] --input_path=<interim_path> --exp_name=<name> --output_dir=<checkpoint_path> ...

Resume any training: add --resume=true to the same command.

3. Start Inference Server

All servers use subcommands: serve for custom checkpoints, or named presets like phail, sim_stack.

# LeRobot 0.4.x SmolVLA — preset (desktop)
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports lerobot-server \
  phail

# LeRobot 0.4.x SmolVLA — custom checkpoint (desktop)
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports lerobot-server \
  --checkpoints_dir=s3://checkpoints/sim_stack/lerobot_04/smolvla_150k/

# LeRobot 0.3.3 ACT (desktop) — requires subcommand
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports lerobot-0_3_3-server \
  serve \
  --checkpoints_dir=s3://checkpoints/sim_stack/lerobot/230226-ee/

# GR00T (desktop or H100) — codec subcommand required
CACHE_ROOT=/home/vertix docker --context desktop compose run --rm --pull always --service-ports groot-server \
  ee_rot6d \
  --checkpoints_dir=s3://checkpoints/sim_stack/groot/ee_rot6d/230226/

# OpenPI (H100) — requires subcommand
docker --context vm-train compose run --rm --pull always --service-ports openpi-server \
  serve \
  --checkpoints_dir=s3://checkpoints/sim_stack/openpi/ee/pi05_positronic_lowmem/230226/

All servers expose WebSocket API on port 8000. Available presets per server:

4. Run Inference Client

# With GUI
uv run positronic-inference sim \
  --policy=.remote --policy.host=desktop --policy.port=8000 \
  --driver.show_gui

# Headless
MUJOCO_GL=egl uv run positronic-inference sim \
  --policy=.remote --policy.host=desktop --policy.port=8000 \
  --driver.show_gui=False --driver.simulation_time=10

Sim Eval End-to-End

# 1. Start server in background (-d flag)
# Check the server script's cfn.cli() to determine if a subcommand is needed (see table above)
CACHE_ROOT=/home/vertix docker --context <machine> compose run -d --rm --pull always --service-ports <server-service> \
  [subcommand] --checkpoints_dir=<checkpoint_path>

# Wait for ready
docker --context <machine> logs --tail 5 <container_id>
# Look for: "Uvicorn running on http://0.0.0.0:8000"

# 2. Run sim episodes
CACHE_ROOT=/home/vertix docker --context <machine> compose run --rm --pull always positronic-inference \
  sim --policy=.remote --policy.host=<server_machine> --policy.port=8000 \
  --driver.num_iterations=50 --driver.simulation_time=30 \
  --output_dir=s3://inference/sim_stack_validation/<run_name>/<model_type>

# 3. View results (locally) — pass top-level dir to compare multiple runs
uv run python -m positronic.cfg.eval sim \
  --dataset.base.path=s3://inference/sim_stack_validation/<run_name> --reset_cache --https
# Opens http://localhost:5001

# 4. Clean up
docker --context <machine> stop <container_id>

Naming: s3://inference/sim_stack_validation/<DDMMYY[-suffix]>/<model_type>/ where model_type is lerobot, groot, openpi, or dreamzero.

Monitoring Background Jobs

grep -o '[0-9]*%' /tmp/claude/-home-vertix-dev-positronic/tasks/<task_id>.output | tail -1
tail -50 /tmp/claude/-home-vertix-dev-positronic/tasks/<task_id>.output
grep -i "error\|complete\|finished" /tmp/claude/-home-vertix-dev-positronic/tasks/<task_id>.output

Common Issues

• CUDA OOM: Each GR00T server uses ~6GB. On 12GB GPUs (desktop), only one server at a time.
• Port conflict: docker ps -a | grep -E "groot-server|openpi-server" then docker stop <id>.
• VM unreachable: ../internal/scripts/start.sh train2 then verify SSH.
• Headless rendering: Use MUJOCO_GL=egl env var.

Nebius Auth (Headless)

1. nebius --no-browser --auth-timeout 5m iam whoami 2>&1 — extract auth URL
2. User clicks URL, browser redirects to http://127.0.0.1:PORT/?code=XXX&state=YYY
3. curl -s "http://127.0.0.1:PORT/?code=XXX&state=YYY" on the machine running nebius
4. Auth completes, VM scripts work

Converted and distributed by TomeVault — claim your Tome and manage your conversions.