Slurm

Slurm

Overview

Slurm (sbatch/squeue/sacct/scontrol) schedules jobs on shared HPC and GPU clusters. Core principle: describe the smallest resource box your job fits in (time, cpus, mem, gpus), throttle your own fan-out so you stay a good neighbor, and infer outcomes from logs/accounting instead of babysitting.

This skill is cluster-agnostic. For a specific site (account format, partition names, GPU models) read that site's docs or run the discovery commands below. For the Digital Research Alliance of Canada, use the digital-research-alliance skill.

First: discover the cluster

Never assume partitions, QoS, or GPU types. Detect them:

sinfo -o "%P %.10l %.6D %G"        # partitions, timelimit, nodes, GRES (GPU types)
sinfo -O partition,gres:40,statelong   # GPU types + node states per partition
sacctmgr show qos format=Name,MaxWall,MaxTRESPU,Priority   # QoS limits
sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES  # what YOU can use
sshare -U                          # your fairshare / current usage
scontrol show config | grep -iE "MaxArraySize|MaxJobCount|SchedulerType"

sbatch script anatomy

#!/bin/bash
#SBATCH --job-name=myrun
#SBATCH --account=def-pi          # billing account (site-specific; required on many clusters)
#SBATCH --partition=gpu           # see `sinfo`
#SBATCH --qos=normal              # see `sacctmgr show qos`
#SBATCH --time=01:00:00           # WALLTIME — shorter = scheduled sooner (backfill)
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G                 # or --mem-per-cpu=8G ; do NOT also set both
#SBATCH --gres=gpu:1              # one GPU (see GPU section)
#SBATCH --output=logs/%x_%j.out   # %x=jobname %j=jobid (see Logging)
#SBATCH --error=logs/%x_%j.err
#SBATCH --mail-type=END,FAIL      # see Email alerts
#SBATCH --mail-user=you@example.com
#SBATCH --signal=B:TERM@120       # SIGTERM 120s before kill → checkpoint gracefully

set -euo pipefail
mkdir -p logs
srun python train.py             # `srun` inside sbatch = proper task launch + accounting

Submit: sbatch job.sh. Print the job id and do not block on it — poll later.

Requesting GPUs — whole, fraction, and shard

• A MIG slice is a hardware-isolated fraction of one GPU (e.g. 1g.5gb, 2g.10gb, 3g.20gb). Use it for dev, inference, or small jobs to stop wasting a whole A100/H100. Profile names vary — discover with sinfo -o "%P %G".
• A shard oversubscribes one physical GPU to several jobs by memory; weaker isolation than MIG. Only available if the site configured it.
• On bundle-ratio clusters a single GPU comes with a fixed CPU/mem share — requesting 1 GPU but 32 cores will queue forever. Match the ratio.
• nvidia-smi, nvtop (interactive), and seff <jobid> (after) tell you if you actually used the GPU. A job that requested a GPU and ran at 0% util is the #1 waste — check it.

Throttling: arrays, concurrency, dependencies

Array jobs run the same script over an index range — the right tool for sweeps:

#SBATCH --array=0-99%8     # 100 tasks, AT MOST 8 running at once  ← the %N throttle

sbatch --array=0-99%8 run.sh           # set throttle at submit
scontrol update ArrayTaskThrottle=4 JobId=12345   # change throttle LIVE on a running array

• %N is the single most important courtesy knob: it caps your concurrent footprint so a big sweep doesn't starve other users (or your own cheap/urgent jobs) against a QoS MaxSubmitJobs/MaxTRESPU cap.
• Dependencies serialize stages: sbatch --dependency=afterok:12345 next.sh (also afterany, afternotok, singleton = one job per name at a time).
• Inside the script, read $SLURM_ARRAY_TASK_ID to pick the work item.

See references/sbatch-template.sh for a full manifest-driven array template.

Pipelines & DAGs (submit the whole graph, then walk away)

The fastest, most hands-off pattern: in ONE session, submit every stage with dependencies wired up, capture the job ids with --parsable, and let Slurm run the DAG without you. Stages start the instant their parents succeed.

# --parsable prints just the jobid (chainable). Build the DAG bottom-up:
prep=$(sbatch --parsable prep.sh)
train=$(sbatch --parsable --dependency=afterok:$prep --array=0-9%4 train.sh)
eval=$(sbatch  --parsable --dependency=afterok:$train eval.sh)        # waits for WHOLE train array
sbatch --dependency=afterok:$eval --mail-type=END report.sh

Dependency types — pick the right edge:

• Combine edges: --dependency=afterok:$a:$b (AND, wait for both); ? separates OR.
• --kill-on-invalid-dep=yes so a child auto-cancels if its parent fails (instead of sitting DependencyNeverSatisfied forever).
• aftercorr + two arrays is the elegant fan-out/fan-in: --array=0-99 prep, then --array=0-99 --dependency=aftercorr:$prep train — 100 independent chains, each link starting as soon as its own parent is done. No barrier.
• Give cheap stages a GPU fraction (MIG/shard) and reserve whole GPUs for the heavy stage, so the DAG packs onto fewer nodes (see GPU section).

Go/no-go gates — stop a sweep before it wastes GPUs

The highest-value DAG pattern: gate an expensive sweep behind a cheap gate experiment that answers "is this even worth running?". Slurm edges key on exit code, so make the gate exit 1 on no-go — the sweep is then auto-cancelled, zero GPU spent.

# Gate: one cheap run that validates a precondition. It must `exit 1` if NO-GO.
#   e.g. gate.sh runs a tiny probe and:  python check.py || exit 1   (nonzero = no-go)
gate=$(sbatch --parsable --job-name=gate --time=00:20:00 --gres=gpu:1 --mem=32G gate.sh)

# Expensive sweep runs ONLY if the gate passed. --kill-on-invalid-dep cancels it
# the instant the gate fails, instead of leaving 100 tasks pending forever.
sbatch --dependency=afterok:$gate --kill-on-invalid-dep=yes \
       --array=0-99%8 --gres=gpu:1 sweep.sh

# Optional: a cheap "why did it fail" report on the no-go branch.
sbatch --dependency=afternotok:$gate --time=00:05:00 nogo_report.sh

The gate's job is to fail fast and cheap. Make check.py exit nonzero when the result says stop — e.g. a baseline that must recover before downstream is valid, a benign run that must train before you spend GPUs attacking it, a smoke metric below threshold, a NaN/divergence check. Chain gates into an escalation ladder (tiny probe → medium probe → full sweep), each afterok the previous, so compute only flows toward configs that keep clearing the bar.

references/dag-pipeline.sh is a complete, runnable DAG submitter (gate → sweep → eval, with an afternotok recovery branch) you adapt and run once over SSH.

Off-cluster autonomous operation

Goal: touch the cluster as little as possible. Pattern:

1. One submission burst. SSH in, run the DAG submitter, capture all ids (--parsable), write a job-id → stage map to a file you own, exit the SSH.
2. Survive preemption (matters on AI/spot partitions): add --requeue, --signal=B:TERM@120 (catch SIGTERM, checkpoint, exit), --open-mode=append (logs survive requeue), and make run.py resume from the latest checkpoint. Now a requeued job continues instead of restarting.
3. Poll, don't babysit. From your laptop/agent: ssh cluster 'sacct -j <ids> --format=JobID,State,Elapsed,ExitCode'. Use the loss-curve rule to scancel revealed-answer jobs. Schedule the next check on an interval (agent: ScheduleWakeup / the loop skill — short while rebalancing, long while everything just runs).
4. Self-healing edges. Wire afternotok cleanup/retry stages into the DAG so a failure handles itself without you logging in.
5. Notify on the boundaries only: --mail-type=END,FAIL,TIME_LIMIT_80 on the final stage and on FAIL of any stage — silence otherwise.

A whole multi-stage sweep can run start-to-finish with a single SSH to submit and a few sacct polls. For fanning the same DAG across several clusters at once, see the digital-research-alliance skill's multi-cluster section.

Email alerts

#SBATCH --mail-type=BEGIN,END,FAIL,REQUEUE,TIME_LIMIT_80
#SBATCH --mail-user=you@example.com
#SBATCH --mail-type=ARRAY_TASKS    # ADD this to get one mail PER array task (else only the array as a whole)

FAIL + TIME_LIMIT_80 (mail at 80% of walltime) is the useful pair: you hear about crashes and about jobs that will hit the wall, without inbox spam. Omit ARRAY_TASKS for big arrays or you self-DOS your inbox.

Logging

#SBATCH --output=logs/%x_%A_%a.out   # array: one file per task
#SBATCH --error=logs/%x_%A_%a.err    # separate stderr (drop to merge into .out)

Best practice: mkdir -p logs in-script (Slurm will NOT create the dir and the job dies silently if it's missing), log the resolved config + git rev-parse HEAD

• hostname/nvidia-smi at the top of every run for reproducibility.

Monitoring (infer the outcome, don't babysit)

Pending Reason= decoded: Priority/Resources = waiting your turn (fine); QOSMaxJobsPerUserLimit = throttle yourself; ReqNodeNotAvail/PartitionConfig = bad request, fix it; AssocGrpGRES = over your GPU allocation.

Loss-curve rule for ML runs: a run that should train and is stuck high, or a run that should be blocked and keeps dropping, has already revealed its answer — scancel it and iterate instead of waiting for walltime.

Control

scancel <id>            scancel <id>_<idx>       scancel -u $USER --state=PENDING
scancel --name=myrun    scancel -u $USER -t PENDING -p gpu   # targeted cancels
scontrol hold <id>      scontrol release <id>    # pause/unpause in queue
scontrol requeue <id>   # restart (use with --requeue + checkpointing)
scontrol update JobId=<id> TimeLimit=02:00:00    # extend walltime (if allowed)
scontrol update JobId=<id> Partition=other       # move a PENDING job

Smoke tests before the big run

Never launch a 100-job sweep untested. In order of cost:

sbatch --test-only job.sh           # validates request + prints est. start; submits NOTHING
salloc --gres=gpu:1 --time=0:30:0   # interactive node: debug live, then exit
srun --gres=gpu:1 --pty bash        # interactive shell on a compute node
sbatch --array=0-0 run.sh           # run ONE array element end-to-end first

Run one element to completion (does it write output? right path? GPU used?) before unleashing --array=0-99%8.

Best-practice repo layout

project/
  slurm/
    job.sbatch              # parameterized; reads $1/env, not hardcoded
    logs/                   # gitignored
  configs/sweep.jsonl       # one JSON per line = one array task
  make_manifest.py          # writes configs/*.jsonl
  run.py                    # reads config by index, idempotent (skip if output exists)

• One sbatch script, many configs via a manifest + array index — not 50 copies.
• Make runs idempotent: check for the output file and exit 0 if it exists, so requeue/resubmit is safe.
• Keep walltime honest and tight: backfill schedules short jobs into gaps sooner.
• export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True for PyTorch GPU jobs.

Common mistakes