Slurm

Slurm

Use a persistent Slurm allocation for environment-sensitive development work, especially ROCm/GPU build, test, profiling, and verification loops. The allocation is held by a salloc shell that lives inside a detached tmux session; every other command (srun, squeue, scancel, builds, tests) runs in a normal shell and addresses the allocation by job ID.

The skill auto-activates when ~/.cache/slurm-detection contains available. The first Slurm skill or hook caller creates that file after checking whether salloc, srun, and squeue are on PATH. An unavailable result makes the skill inactive, so the agent runs commands directly without an srun wrapper

• and without adding any Slurm-related text to the final report.

Quick Start

1. Confirm the execution boundary:
   • Read-only exploration may happen outside Slurm.
   • Cursor file edits are allowed.
   • Builds, tests, profiler runs, GPU checks, and final verification commands run inside the Slurm allocation via srun --jobid="$JOBID" <command>.
2. Read or initialize the cached Slurm detection result, then prepare the allocation. This single block reuses an existing cursor-agent allocation if one is exported in the environment, and otherwise detects-or-creates one in tmux:

SLURM_DETECTION_CACHE="$HOME/.cache/slurm-detection"
SLURM_DETECTION_STATE=""
if [[ -r "$SLURM_DETECTION_CACHE" ]]; then
  SLURM_DETECTION_STATE=$(<"$SLURM_DETECTION_CACHE")
fi

if [[ "$SLURM_DETECTION_STATE" != "available" \
   && "$SLURM_DETECTION_STATE" != "unavailable" ]]; then
  rm -f "$SLURM_DETECTION_CACHE" 2>/dev/null || true

  if command -v salloc >/dev/null 2>&1 \
     && command -v srun   >/dev/null 2>&1 \
     && command -v squeue >/dev/null 2>&1; then
    SLURM_DETECTION_STATE="available"
  else
    SLURM_DETECTION_STATE="unavailable"
  fi

  if mkdir -p "$HOME/.cache" 2>/dev/null; then
    SLURM_DETECTION_TMP=$(mktemp "$HOME/.cache/.slurm-detection.XXXXXX" 2>/dev/null || true)
    if [[ -n "$SLURM_DETECTION_TMP" ]]; then
      printf '%s\n' "$SLURM_DETECTION_STATE" > "$SLURM_DETECTION_TMP"
      chmod 600 "$SLURM_DETECTION_TMP" 2>/dev/null || true
      if ! ln "$SLURM_DETECTION_TMP" "$SLURM_DETECTION_CACHE" 2>/dev/null \
         && [[ -r "$SLURM_DETECTION_CACHE" ]]; then
        SLURM_CACHED_STATE=$(<"$SLURM_DETECTION_CACHE")
        if [[ "$SLURM_CACHED_STATE" == "available" \
           || "$SLURM_CACHED_STATE" == "unavailable" ]]; then
          SLURM_DETECTION_STATE="$SLURM_CACHED_STATE"
        fi
      fi
      rm -f "$SLURM_DETECTION_TMP"
    fi
  fi
fi

if [[ "$SLURM_DETECTION_STATE" != "available" ]]; then
  # Cached Slurm detection says the skill is inactive.
  # The skill does not apply: run commands directly, no srun wrapper,
  # no Slurm-related text in the final report.
  return 0 2>/dev/null || exit 0
fi

JOBID=""
if [ -n "${SLURM_JOB_ID:-}" ] \
   && [ "$(squeue -j "$SLURM_JOB_ID" -h -o '%j' 2>/dev/null)" = "cursor-agent" ]; then
  JOBID="$SLURM_JOB_ID"
fi

if [ -z "$JOBID" ]; then
  JOBID=$(squeue -u "$USER" -h -o "%A" -t R --name=cursor-agent | head -n1)

  if [ -z "$JOBID" ]; then
    GPU_PREFS=(
      "gfx942-mi300x"
      "gfx942-mi308x"
      "gfx942-mi300a"
      "gfx942-mi325x"
      "gfx950-mi350x"
      "gfx950-mi355x"
    )
    GRES_NAME=""
    for gpu in "${GPU_PREFS[@]}"; do
      if sinfo -h -N -o "%G %t" 2>/dev/null \
          | awk -v g="$gpu" '$1 ~ g && ($2 == "idle" || $2 == "mix") { found=1; exit }
                              END { exit !found }'; then
        GRES_NAME="$gpu"
        break
      fi
    done
    [ -z "$GRES_NAME" ] && GRES_NAME="gfx942-mi300x"
    GRES="gpu:${GRES_NAME}:1"

    tmux new-session -d -s cursor-agent-alloc \
      "salloc --job-name=cursor-agent \
              --gres=$GRES \
              --cpus-per-task=32 \
              --mem=1T \
              --time=12:00:00"
    for _ in $(seq 1 60); do
      JOBID=$(squeue -u "$USER" -h -o "%A" -t R --name=cursor-agent | head -n1)
      [ -n "$JOBID" ] && break
      sleep 5
    done
  fi
fi

3. Run each command from a regular shell using the captured $JOBID (do not wrap these in tmux):

srun --jobid="$JOBID" rocminfo
srun --jobid="$JOBID" python -m pytest path/to/test_file.py -k test_name

4. Capture evidence in timestamped logs by wrapping the srun invocation (not the salloc/tmux line) with tee.
5. Run targeted verification first, then broader build/install/tests when the change touches shared or user-facing behavior.

Explicit standalone fallback

The persistent cursor-agent allocation above is the only normal Cursor execution path. A human running a GPU-dependent skill helper directly outside Cursor may explicitly request one short-lived allocation:

bash <SLURM_SKILL_DIR>/scripts/run-standalone.sh -- <command> [args...]

The wrapper executes directly when already inside a Slurm step, reuses an exported cursor-agent allocation through srun, or creates one short-lived salloc ... srun allocation. When Slurm commands are unavailable it executes inline. It returns the command's status and is never called automatically by sibling skills. See REFERENCE.md.

Behavior Rules

• Read Slurm availability from ~/.cache/slurm-detection. On the first call, or when the file is empty or invalid, check salloc, srun, and squeue and atomically cache available or unavailable. Never source the file. Both results persist until the file is deleted manually. If the cache cannot be written, use the detected result for the current invocation. When the result is unavailable, do not wrap commands with srun and do not add a "Slurm unavailable" note to the report.
• If $SLURM_STEP_ID is set, run directly inside that step. If only $SLURM_JOB_ID is set, reuse it when squeue -j "$SLURM_JOB_ID" -h -o '%j' equals cursor-agent. Reject a differently named inherited job rather than nesting another allocation inside it.
• Use the baseline salloc flags --job-name=cursor-agent --gres=$GRES --cpus-per-task=32 --mem=1T --time=12:00:00. $GRES is selected automatically by probing sinfo for the first preferred GPU type with idle or mix capacity. The preference order is mi300x, mi308x, mi300a, mi325x (gfx942 family), then mi350x, mi355x (gfx950 family). If nothing is idle anywhere, queue on gpu:gfx942-mi300x:1. Ask for site-specific additions (partition, account, QoS) only if these defaults fail.
• Run only the salloc launch inside tmux new-session -d -s cursor-agent-alloc .... Do not wrap srun, squeue, scancel, the polling loop, or any verification command in tmux.
• Whenever the skill is active, runtime-sensitive work (builds, tests, profilers, GPU/ROCm checks) goes through srun --jobid="$JOBID"; only file inspection, planning, and source edits run on the login node.
• Do not silently piggyback on a job whose name is not cursor-agent - it may have different --gres/time and the agent did not pick its flags.
• Keep final reports concise: command, pass/fail status, key output summary, and log location.
• If squeue -j "$JOBID" shows the job gone (time limit, scancel, or tmux session killed), re-run the detect-or-create step and rerun only missing verification.
• Do not copy salloc logic into sibling helpers. Outside Cursor, a user may explicitly wrap one command with scripts/run-standalone.sh; inside Cursor, use the persistent allocation workflow.

Additional Resources

• REFERENCE.md
• EXAMPLES.md
• scripts/run-standalone.sh