# Runai

> Use when submitting, monitoring, exec-ing into, or troubleshooting RunAI GPU jobs for the sleap-roots pipeline on the Salk cluster (project busch-lab / namespace runai-busch-lab) — e.g. running the predictor stage interactively, debugging a stuck pod, or staging data over the /hpi/hpi_dev NFS mount. Adapted from the mosquito-cfd runai-cluster-skill.

- Skill: `talmolab/runai` (Agent Skill)
- Install (CLI): `npx skillmds@latest add talmolab/runai`
- Raw SKILL.md: https://api.skillmd.com/api/skills/talmolab/runai/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: talmolab (https://skillmd.com/u/talmolab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/talmolab/runai

---


# RunAI cluster skill — sleap-roots-pipeline

RunAI CLI v2 assistance for the Salk GPU cluster. The **production path is Argo**
(`runai_run_pipeline.sh` → `argo submit`); use the `runai workspace` CLI here for the
**interactive / ad-hoc / debug** path — running a single stage by hand, staging data,
exec-ing into a live pod, or diagnosing scheduling.

> Project: **`busch-lab`** · Namespace: **`runai-busch-lab`** · Cluster Argo server:
> `gpu-master:8888`. **Updated 2026-08-13 — this pipeline now targets busch-lab only** (was
> `talmo-lab`; see `sleap-roots-pipeline.yaml`'s `metadata.namespace`). `talmo-lab`/
> `runai-talmo-lab` remain live on the cluster (20 GPU quota) but are no longer this pipeline's
> target — don't assume examples elsewhere still apply without checking. If a manifest still
> says `tye-lab`, it is stale — fix it.

## 1. WSL command execution pattern

RunAI runs in **WSL**, not Windows PowerShell, and needs an explicit KUBECONFIG:

```bash
wsl -e bash -c "export KUBECONFIG=~/.kube/kubeconfig-runai-busch-lab-argo-user.yaml && \
  runai <command>"
```

- `runai` is assumed on `PATH`; if your install isn't, use its absolute path (e.g.
  `"$HOME/.runai/bin/runai"`). **Verify the `KUBECONFIG` path and the `runai` binary location
  in your own WSL environment** before constructing commands — these are operator-specific.
- In **Git Bash** (not WSL), prefix cluster-path commands with `MSYS_NO_PATHCONV=1` to stop
  `/hpi/...` from being mangled into a Windows path.

## 2. Path mapping (Windows ↔ WSL ↔ cluster)

| Context | Path |
|---|---|
| Windows (local) | `Z:\users\eberrigan\...` |
| WSL | `/mnt/hpi_dev/users/eberrigan/...` |
| Cluster (NFS, in `--host-path` + container mounts) | `/hpi/hpi_dev/users/eberrigan/...` |

`Z:` = `\\multilab-na.ad.salk.edu\hpi_dev` (Salk VPN / multilab-na). Host-path mount syntax:

```
--host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/<dir>,mount=/data,mount-propagation=HostToContainer,readwrite
```

## 3. Workspace lifecycle

All commands take `-p busch-lab`. Use **`runai workspace`** (does not auto-terminate — you
clean up manually) rather than `runai training` (auto-terminates on completion).

| Operation | Command |
|---|---|
| List | `runai workspace list -p busch-lab` |
| Describe | `runai workspace describe <name> -p busch-lab` |
| Logs | `runai workspace logs <name> -p busch-lab --follow` |
| Exec | `runai workspace exec <name> -p busch-lab -- <cmd>` |
| Interactive shell | `runai workspace exec <name> -p busch-lab --stdin --tty -- /bin/bash` |
| Delete | `runai workspace delete <name> -p busch-lab` |

> Use `runai workspace exec`, **not** `kubectl exec` — RunAI manages its own auth layer.

## 4. Resource flags (CLI v2)

| Need | Flag |
|---|---|
| GPU (whole) | `--gpu-devices-request 1` |
| GPU (fractional, relative) | `--gpu-portion-request 0.5` (fraction of a GPU, 0-1) |
| GPU (fractional, absolute) | `--gpu-memory-request 8192M` (absolute amount, e.g. `1G`/`500M` — the predictor template annotates the pod-level `gpu-memory: "8192"` (MiB); using `8192M` here rather than `8G` since a bare `G` suffix may mean decimal `10^9` bytes elsewhere in this CLI, ~7% less than `8192` MiB/`8Gi` — this hasn't been exercised live to confirm which convention `--gpu-memory-request` actually follows, so `M` avoids the ambiguity rather than resolving it) |
| CPU cores | `--cpu-core-request 12` |
| Memory | `--cpu-memory-request 32G` |
| Always re-pull image | `--image-pull-policy Always` |

Only the **predictor** stage needs a GPU; `models-downloader` and `trait-extractor` are
CPU-only. The predictor template uses a **pod-level** `gpu-memory: "8192"` annotation with **no**
`nvidia.com/gpu` resource (fixed in
[issue #25](https://github.com/talmolab/sleap-roots-pipeline/issues/25) — it previously pinned an
inert *object-level* `gpu-fraction: "0.5"` annotation alongside a hard `nvidia.com/gpu: 1`, which
silently claimed a whole GPU regardless of the annotation). Annotation placement matters: only
`spec.templates[].metadata.annotations` (pod-level) is copied onto the pod by Argo — the
WorkflowTemplate object's own `metadata.annotations` (top of the file) never is.

## 5. Stage images

Current registry is **GitLab** (`registry.gitlab.com/salk-tm/...`); the roadmap A0 target is to
migrate these to GHCR, not yet done — so use the GitLab refs until then.

| Stage | Image |
|---|---|
| models-downloader | `registry.gitlab.com/salk-tm/models-downloader:<tag>` |
| predictor (GPU) | `registry.gitlab.com/salk-tm/sleap-roots-predict:<tag>` |
| trait-extractor | `registry.gitlab.com/salk-tm/sleap-roots-traits:<tag>` |

Pin a tag/digest — never `:latest`. Confirm the tag exists in the registry before submitting.

## 6. Example — run the predictor stage interactively

The predictor reads three container dirs — `/workspace/images_input`, `/workspace/models_input`,
`/workspace/output` — which are also its entrypoint's positional args. **Mount the host dirs
to those exact container paths.** Note the non-obvious remap: the *models-downloader output*
dir (`models_downloader_output`) is what feeds the predictor's `models_input`.

```bash
wsl -e bash -c "export KUBECONFIG=~/.kube/kubeconfig-runai-busch-lab-argo-user.yaml && \
runai workspace submit srp-predict-test \
  -p busch-lab \
  --image registry.gitlab.com/salk-tm/sleap-roots-predict:<tag> \
  --image-pull-policy Always \
  --gpu-memory-request 8192M \
  --cpu-core-request 8 \
  --cpu-memory-request 16G \
  --host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/images_downloader_output,mount=/workspace/images_input,mount-propagation=HostToContainer \
  --host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/models_downloader_output,mount=/workspace/models_input,mount-propagation=HostToContainer \
  --host-path path=/hpi/hpi_dev/users/eberrigan/<dataset>/predictions,mount=/workspace/output,mount-propagation=HostToContainer,readwrite \
  -- bash -c '<predict entrypoint> /workspace/images_input /workspace/models_input /workspace/output; sleep infinity'"
```

`sleep infinity` keeps the pod alive after the run so you can `runai workspace exec` in to
inspect outputs. **Delete it when done** (`runai workspace delete srp-predict-test -p busch-lab`).

## 7. Preemptibility & GPU over-quota

**Preemptibility is set by `priorityClassName`, NOT by the `preemptible: "true"` annotation**
the templates carry (that annotation is a UI/convention breadcrumb only). Run:ai treats
`priorityClassName` **≥ 100 as non-preemptible**, **< 100 as preemptible**:

| Class | Preemptible? | Behaviour |
|---|---|---|
| `very-high` (150) | no | **default when `priorityClassName` is unset on this cluster** — do not rely on omitting the field as a "safe default"; it's the most aggressive non-preemptible tier here, not a neutral one |
| `high` (125), `build` (100) | no | must fit the project's **deserved quota**; never evicted |
| `interactive-preemptible` (75), `train` (50) | yes | may use **over-quota** GPUs; may be evicted → pair with `retryStrategy` |

Cluster-admin-confirmed naming (2026-08-06): the 125 tier's real name on this cluster is
**`high`**, not `inference` — corrected here after an earlier assumption. The predictor template
uses `high` (set 2026-08-06, per cluster-admin guidance, since trait-extractor has no
skip-if-done yet — see issue #37 — so avoiding eviction-triggered whole-batch recomputation
outweighs bursting above quota for now). The other three stage templates
(images-downloader/trait-extractor/write-back) stay on **`interactive-preemptible`** —
**never remove that field outright**, since an unset `priorityClassName` lands at `very-high`
(150) on this cluster, not something safer. The predictor's GPU jobs typically run *within*
quota, so over-quota preemption isn't usually exercised — but if a GPU
pod is stuck `Pending`/`Unschedulable` with:

```
NonPreemptibleOverQuota: Non-preemptible workload is over quota. ... busch-lab quota is 2 GPUs,
while 2 GPUs are already allocated for non-preemptible pods. Use a preemptible workload to go over quota.
```

busch-lab's deserved quota is only **2 GPUs** (vs. talmo-lab's 20) — over-quota scheduling is far
more likely to actually happen here. Check current usage before submitting anything
non-preemptible (`kubectl get pods -n runai-busch-lab` — other jobs holding whole GPUs, not just
fractional ones, will block a fractional predictor pod from landing even though the memory math
looks fine).

set the priority class:

- **Argo** WorkflowTemplate/Workflow: `spec.templates[].priorityClassName: interactive-preemptible`
- **`runai` CLI**: submit as a **training** workload (preemptible) instead of a workspace.

## 8. Troubleshooting

| Symptom | Fix |
|---|---|
| Auth error / token expired | `runai login remote-browser` (then `runai whoami`) |
| Job stuck `Pending` | check cluster capacity + resource requests (`runai workspace describe`); if `NonPreemptibleOverQuota`, see §7 |
| Mount error at startup | verify `--host-path` syntax and that the `/hpi/hpi_dev/...` directory exists on the node |
| `ImagePullBackOff` | confirm the `registry.gitlab.com/salk-tm/...` tag exists; test `docker pull` of the same tag |
| `gh` returns HTTP 403 | `unset GITHUB_TOKEN` first (long-lived fine-grained tokens are blocked by the `talmolab` org) |
| Git Bash mangles `/hpi/...` | prefix with `MSYS_NO_PATHCONV=1` (or run in WSL) |

## 9. CLI v1 → v2 migration

| v1 (deprecated) | v2 (current) |
|---|---|
| `runai submit` | `runai workspace submit` |
| `runai list jobs` | `runai workspace list` |
| `runai describe job` | `runai workspace describe` |
| `runai logs` | `runai workspace logs` |
| `runai delete job` | `runai workspace delete` |
| `--cpu 12` | `--cpu-core-request 12` |
| `--memory 32G` | `--cpu-memory-request 32G` |
| `--gpu 1` | `--gpu-devices-request 1` |
| `--host-path /src:/dst:ro` | `--host-path path=/src,mount=/dst,mount-propagation=HostToContainer` |

---

*Adapted from the `mosquito-cfd` `runai-cluster-skill` (CFD/IAMReX → sleap-roots
predict/traits). The same Salk cluster and WSL/KUBECONFIG pattern apply (project is
`busch-lab` here, not `mosquito-cfd`'s); workloads, images, and the GPU-on-predictor-only shape
are sleap-roots-specific.*