Runai

RunAI cluster skill — sleap-roots-pipeline

RunAI CLI v2 assistance for the Salk GPU cluster. The production path is Argo (runai_run_pipeline.sh → argo submit); use the runai workspace CLI here for the interactive / ad-hoc / debug path — running a single stage by hand, staging data, exec-ing into a live pod, or diagnosing scheduling.

Project: busch-lab · Namespace: runai-busch-lab · Cluster Argo server: gpu-master:8888. Updated 2026-08-13 — this pipeline now targets busch-lab only (was talmo-lab; see sleap-roots-pipeline.yaml's metadata.namespace). talmo-lab/ runai-talmo-lab remain live on the cluster (20 GPU quota) but are no longer this pipeline's target — don't assume examples elsewhere still apply without checking. If a manifest still says tye-lab, it is stale — fix it.

1. WSL command execution pattern

RunAI runs in WSL, not Windows PowerShell, and needs an explicit KUBECONFIG:

wsl -e bash -c "export KUBECONFIG=~/.kube/kubeconfig-runai-busch-lab-argo-user.yaml && \
  runai <command>"

• runai is assumed on PATH; if your install isn't, use its absolute path (e.g. "$HOME/.runai/bin/runai"). Verify the KUBECONFIG path and the runai binary location in your own WSL environment before constructing commands — these are operator-specific.
• In Git Bash (not WSL), prefix cluster-path commands with MSYS_NO_PATHCONV=1 to stop /hpi/... from being mangled into a Windows path.

2. Path mapping (Windows ↔ WSL ↔ cluster)

Z: = \\multilab-na.ad.salk.edu\hpi_dev (Salk VPN / multilab-na). Host-path mount syntax:

--host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/<dir>,mount=/data,mount-propagation=HostToContainer,readwrite

3. Workspace lifecycle

All commands take -p busch-lab. Use runai workspace (does not auto-terminate — you clean up manually) rather than runai training (auto-terminates on completion).

Use runai workspace exec, not kubectl exec — RunAI manages its own auth layer.

4. Resource flags (CLI v2)

Only the predictor stage needs a GPU; models-downloader and trait-extractor are CPU-only. The predictor template uses a pod-level gpu-memory: "8192" annotation with no nvidia.com/gpu resource (fixed in issue #25 — it previously pinned an inert object-level gpu-fraction: "0.5" annotation alongside a hard nvidia.com/gpu: 1, which silently claimed a whole GPU regardless of the annotation). Annotation placement matters: only spec.templates[].metadata.annotations (pod-level) is copied onto the pod by Argo — the WorkflowTemplate object's own metadata.annotations (top of the file) never is.

5. Stage images

Current registry is GitLab (registry.gitlab.com/salk-tm/...); the roadmap A0 target is to migrate these to GHCR, not yet done — so use the GitLab refs until then.

Pin a tag/digest — never :latest. Confirm the tag exists in the registry before submitting.

6. Example — run the predictor stage interactively

The predictor reads three container dirs — /workspace/images_input, /workspace/models_input, /workspace/output — which are also its entrypoint's positional args. Mount the host dirs to those exact container paths. Note the non-obvious remap: the models-downloader output dir (models_downloader_output) is what feeds the predictor's models_input.

wsl -e bash -c "export KUBECONFIG=~/.kube/kubeconfig-runai-busch-lab-argo-user.yaml && \
runai workspace submit srp-predict-test \
  -p busch-lab \
  --image registry.gitlab.com/salk-tm/sleap-roots-predict:<tag> \
  --image-pull-policy Always \
  --gpu-memory-request 8192M \
  --cpu-core-request 8 \
  --cpu-memory-request 16G \
  --host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/images_downloader_output,mount=/workspace/images_input,mount-propagation=HostToContainer \
  --host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/models_downloader_output,mount=/workspace/models_input,mount-propagation=HostToContainer \
  --host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/predictions,mount=/workspace/output,mount-propagation=HostToContainer,readwrite \
  -- bash -c '<predict entrypoint> /workspace/images_input /workspace/models_input /workspace/output; sleep infinity'"

sleep infinity keeps the pod alive after the run so you can runai workspace exec in to inspect outputs. Delete it when done (runai workspace delete srp-predict-test -p busch-lab).

7. Preemptibility & GPU over-quota

Preemptibility is set by priorityClassName, NOT by the preemptible: "true" annotation the templates carry (that annotation is a UI/convention breadcrumb only). Run:ai treats priorityClassName ≥ 100 as non-preemptible, < 100 as preemptible:

Cluster-admin-confirmed naming (2026-08-06): the 125 tier's real name on this cluster is high, not inference — corrected here after an earlier assumption. The predictor template uses high (set 2026-08-06, per cluster-admin guidance, since trait-extractor has no skip-if-done yet — see issue #37 — so avoiding eviction-triggered whole-batch recomputation outweighs bursting above quota for now). The other three stage templates (images-downloader/trait-extractor/write-back) stay on interactive-preemptible — never remove that field outright, since an unset priorityClassName lands at very-high (150) on this cluster, not something safer. The predictor's GPU jobs typically run within quota, so over-quota preemption isn't usually exercised — but if a GPU pod is stuck Pending/Unschedulable with:

NonPreemptibleOverQuota: Non-preemptible workload is over quota. ... busch-lab quota is 2 GPUs,
while 2 GPUs are already allocated for non-preemptible pods. Use a preemptible workload to go over quota.

busch-lab's deserved quota is only 2 GPUs (vs. talmo-lab's 20) — over-quota scheduling is far more likely to actually happen here. Check current usage before submitting anything non-preemptible (kubectl get pods -n runai-busch-lab — other jobs holding whole GPUs, not just fractional ones, will block a fractional predictor pod from landing even though the memory math looks fine).

set the priority class:

• Argo WorkflowTemplate/Workflow: spec.templates[].priorityClassName: interactive-preemptible
• runai CLI: submit as a training workload (preemptible) instead of a workspace.

8. Troubleshooting

9. CLI v1 → v2 migration

Adapted from the mosquito-cfd runai-cluster-skill (CFD/IAMReX → sleap-roots predict/traits). The same Salk cluster and WSL/KUBECONFIG pattern apply (project is busch-lab here, not mosquito-cfd's); workloads, images, and the GPU-on-predictor-only shape are sleap-roots-specific.