# Slurm

> ALWAYS use this skill on every task that runs a shell command. Runs development, build, test, and verification workflows through an interactive Slurm allocation (salloc in detached tmux, then per-command srun --jobid) while preserving Cursor editing ergonomics. Reads cached availability from ~/.cache/slurm-detection and silently does not apply when Slurm is unavailable.

- Skill: `abchoudh-amd/slurm` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add abchoudh-amd/slurm`
- Raw SKILL.md: https://api.skillmd.com/api/skills/abchoudh-amd/slurm/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: abchoudh-amd (https://skillmd.com/u/abchoudh-amd)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/abchoudh-amd/slurm

---


# Slurm

Use a persistent Slurm allocation for environment-sensitive development work,
especially ROCm/GPU build, test, profiling, and verification loops. The
allocation is held by a `salloc` shell that lives inside a detached tmux
session; every other command (`srun`, `squeue`, `scancel`, builds, tests) runs
in a normal shell and addresses the allocation by job ID.

The skill auto-activates when `~/.cache/slurm-detection` contains `available`.
The first Slurm skill or hook caller creates that file after checking whether
`salloc`, `srun`, and `squeue` are on `PATH`. An `unavailable` result makes the
skill inactive, so the agent runs commands directly without an `srun` wrapper
- and without adding any Slurm-related text to the final report.

## Quick Start

1. Confirm the execution boundary:
   - Read-only exploration may happen outside Slurm.
   - Cursor file edits are allowed.
   - Builds, tests, profiler runs, GPU checks, and final verification commands
     run inside the Slurm allocation via `srun --jobid="$JOBID" <command>`.
2. Read or initialize the cached Slurm detection result, then prepare the
   allocation. This single block reuses an existing `cursor-agent` allocation
   if one is exported in the environment, and otherwise detects-or-creates one
   in tmux:

```bash
SLURM_DETECTION_CACHE="$HOME/.cache/slurm-detection"
SLURM_DETECTION_STATE=""
if [[ -r "$SLURM_DETECTION_CACHE" ]]; then
  SLURM_DETECTION_STATE=$(<"$SLURM_DETECTION_CACHE")
fi

if [[ "$SLURM_DETECTION_STATE" != "available" \
   && "$SLURM_DETECTION_STATE" != "unavailable" ]]; then
  rm -f "$SLURM_DETECTION_CACHE" 2>/dev/null || true

  if command -v salloc >/dev/null 2>&1 \
     && command -v srun   >/dev/null 2>&1 \
     && command -v squeue >/dev/null 2>&1; then
    SLURM_DETECTION_STATE="available"
  else
    SLURM_DETECTION_STATE="unavailable"
  fi

  if mkdir -p "$HOME/.cache" 2>/dev/null; then
    SLURM_DETECTION_TMP=$(mktemp "$HOME/.cache/.slurm-detection.XXXXXX" 2>/dev/null || true)
    if [[ -n "$SLURM_DETECTION_TMP" ]]; then
      printf '%s\n' "$SLURM_DETECTION_STATE" > "$SLURM_DETECTION_TMP"
      chmod 600 "$SLURM_DETECTION_TMP" 2>/dev/null || true
      if ! ln "$SLURM_DETECTION_TMP" "$SLURM_DETECTION_CACHE" 2>/dev/null \
         && [[ -r "$SLURM_DETECTION_CACHE" ]]; then
        SLURM_CACHED_STATE=$(<"$SLURM_DETECTION_CACHE")
        if [[ "$SLURM_CACHED_STATE" == "available" \
           || "$SLURM_CACHED_STATE" == "unavailable" ]]; then
          SLURM_DETECTION_STATE="$SLURM_CACHED_STATE"
        fi
      fi
      rm -f "$SLURM_DETECTION_TMP"
    fi
  fi
fi

if [[ "$SLURM_DETECTION_STATE" != "available" ]]; then
  # Cached Slurm detection says the skill is inactive.
  # The skill does not apply: run commands directly, no srun wrapper,
  # no Slurm-related text in the final report.
  return 0 2>/dev/null || exit 0
fi

JOBID=""
if [ -n "${SLURM_JOB_ID:-}" ] \
   && [ "$(squeue -j "$SLURM_JOB_ID" -h -o '%j' 2>/dev/null)" = "cursor-agent" ]; then
  JOBID="$SLURM_JOB_ID"
fi

if [ -z "$JOBID" ]; then
  JOBID=$(squeue -u "$USER" -h -o "%A" -t R --name=cursor-agent | head -n1)

  if [ -z "$JOBID" ]; then
    GPU_PREFS=(
      "gfx942-mi300x"
      "gfx942-mi308x"
      "gfx942-mi300a"
      "gfx942-mi325x"
      "gfx950-mi350x"
      "gfx950-mi355x"
    )
    GRES_NAME=""
    for gpu in "${GPU_PREFS[@]}"; do
      if sinfo -h -N -o "%G %t" 2>/dev/null \
          | awk -v g="$gpu" '$1 ~ g && ($2 == "idle" || $2 == "mix") { found=1; exit }
                              END { exit !found }'; then
        GRES_NAME="$gpu"
        break
      fi
    done
    [ -z "$GRES_NAME" ] && GRES_NAME="gfx942-mi300x"
    GRES="gpu:${GRES_NAME}:1"

    tmux new-session -d -s cursor-agent-alloc \
      "salloc --job-name=cursor-agent \
              --gres=$GRES \
              --cpus-per-task=32 \
              --mem=1T \
              --time=12:00:00"
    for _ in $(seq 1 60); do
      JOBID=$(squeue -u "$USER" -h -o "%A" -t R --name=cursor-agent | head -n1)
      [ -n "$JOBID" ] && break
      sleep 5
    done
  fi
fi
```

3. Run each command from a regular shell using the captured `$JOBID` (do **not**
   wrap these in tmux):

```bash
srun --jobid="$JOBID" rocminfo
srun --jobid="$JOBID" python -m pytest path/to/test_file.py -k test_name
```

4. Capture evidence in timestamped logs by wrapping the `srun` invocation
   (not the salloc/tmux line) with `tee`.
5. Run targeted verification first, then broader build/install/tests when the
   change touches shared or user-facing behavior.

## Explicit standalone fallback

The persistent `cursor-agent` allocation above is the only normal Cursor
execution path. A human running a GPU-dependent skill helper directly outside
Cursor may explicitly request one short-lived allocation:

```bash
bash <SLURM_SKILL_DIR>/scripts/run-standalone.sh -- <command> [args...]
```

The wrapper executes directly when already inside a Slurm step, reuses an
exported `cursor-agent` allocation through `srun`, or creates one short-lived
`salloc ... srun` allocation. When Slurm commands are unavailable it executes
inline. It returns the command's status and is never called automatically by
sibling skills. See [REFERENCE.md](REFERENCE.md#explicit-standalone-execution).

## Behavior Rules

- Read Slurm availability from `~/.cache/slurm-detection`. On the first call,
  or when the file is empty or invalid, check `salloc`, `srun`, and `squeue`
  and atomically cache `available` or `unavailable`. Never source the file.
  Both results persist until the file is deleted manually. If the cache cannot
  be written, use the detected result for the current invocation. When the
  result is `unavailable`, do not wrap commands with `srun` and do not add a
  "Slurm unavailable" note to the report.
- If `$SLURM_STEP_ID` is set, run directly inside that step. If only
  `$SLURM_JOB_ID` is set, reuse it when `squeue -j "$SLURM_JOB_ID" -h -o
  '%j'` equals `cursor-agent`. Reject a differently named inherited job rather
  than nesting another allocation inside it.
- Use the baseline `salloc` flags `--job-name=cursor-agent --gres=$GRES
  --cpus-per-task=32 --mem=1T --time=12:00:00`. `$GRES` is selected
  automatically by probing `sinfo` for the first preferred GPU type with
  `idle` or `mix` capacity. The preference order is `mi300x`, `mi308x`,
  `mi300a`, `mi325x` (gfx942 family), then `mi350x`, `mi355x` (gfx950
  family). If nothing is idle anywhere, queue on `gpu:gfx942-mi300x:1`. Ask
  for site-specific additions (partition, account, QoS) only if these
  defaults fail.
- Run only the `salloc` launch inside `tmux new-session -d -s
  cursor-agent-alloc ...`. Do not wrap `srun`, `squeue`, `scancel`, the
  polling loop, or any verification command in tmux.
- Whenever the skill is active, runtime-sensitive work (builds, tests,
  profilers, GPU/ROCm checks) goes through `srun --jobid="$JOBID"`; only file
  inspection, planning, and source edits run on the login node.
- Do not silently piggyback on a job whose name is not `cursor-agent` - it may
  have different `--gres`/time and the agent did not pick its flags.
- Keep final reports concise: command, pass/fail status, key output summary, and
  log location.
- If `squeue -j "$JOBID"` shows the job gone (time limit, scancel, or tmux
  session killed), re-run the detect-or-create step and rerun only missing
  verification.
- Do not copy `salloc` logic into sibling helpers. Outside Cursor, a user may
  explicitly wrap one command with `scripts/run-standalone.sh`; inside Cursor,
  use the persistent allocation workflow.

## Additional Resources

- [REFERENCE.md](REFERENCE.md)
- [EXAMPLES.md](EXAMPLES.md)
- [scripts/run-standalone.sh](scripts/run-standalone.sh)