Slurm Jobs

SLURM for deep learning training

Assumes Linux compute nodes and a PyTorch stack. SLURM has no Windows client — drive it over ssh (from WSL or Windows Terminal); write scripts with LF endings or sbatch fails with /bin/bash^M: bad interpreter.

Cheat sheet: the commands you actually use

sbatch train.sh                      # submit; prints "Submitted batch job 12345"
sbatch --parsable train.sh           # prints just "12345" — use in scripts
sbatch --test-only train.sh          # validate directives + estimated start, submit nothing

squeue -u $USER -o "%.10i %.12P %.24j %.2t %.10M %.10l %.4D %R"   # %R = REASON, the useful column
squeue --start -j 12345              # backfill's estimated start time (only if backfill sched)
squeue -u $USER -t PD                # only pending

scancel 12345                        # kill job;  12345_7 kills one array task
scancel -u $USER -t PENDING          # kill all your pending jobs, leave running ones alone
scancel -b --signal=USR1 12345       # hand-trigger your checkpoint-and-requeue handler.
                                     # -b/--batch is REQUIRED: without it the signal goes to the
                                     # job steps, not the batch shell where your trap lives.

scontrol show job 12345              # live truth: Reason, TRES, StartTime, NodeList, StdOut
scontrol show hostnames "$SLURM_JOB_NODELIST"   # expands "gpu[03-06]" -> one host per line
scontrol requeue 12345               # put a running job back in the queue (needs --requeue)
scontrol show config | grep -E "MaxArraySize|MaxJobCount|DefMemPerCPU"

sinfo -o "%.20P %.5a %.12l %.6D %.6t %N"        # partitions, time limits, node states
sinfo -p gpu --Node -o "%N %t %C %m %G"         # per-node: state, CPU A/I/O/T, mem, GRES
sinfo -R                                        # why nodes are drained

seff 12345                           # after the fact: CPU% and Mem% of what you requested
sacct -j 12345 --units=G -o JobID,JobName%20,State,ExitCode,Elapsed,Timelimit,MaxRSS,ReqMem,AllocTRES%40
sacct -X -u $USER -S 2026-08-01 -o JobID,JobName%24,State,Elapsed,NodeList   # -X = job rows only

sacct reads the accounting DB, so it is the only tool that works after a job ends. squeue/scontrol show job only know about live and very recently finished jobs.

Interactive / debug sessions

# One-shot interactive shell on a GPU node (dies when you disconnect):
srun --partition=gpu --gres=gpu:1 --cpus-per-task=8 --mem=32G --time=01:00:00 --pty bash -l

# Allocation you can attach multiple steps to:
salloc --partition=gpu --gres=gpu:2 --cpus-per-task=16 --mem=64G --time=02:00:00
# ...then, inside the allocation shell:
srun --pty bash -l          # shell ON the compute node (salloc's shell may be on the login node)

Use bash -l or your module/conda hooks are not loaded. Run srun --pty inside tmux on the login node — the moment your ssh drops, an untmuxed interactive job dies.

The resource directives, and the CPU trap

#SBATCH --job-name=segformer-pretrain
#SBATCH --partition=gpu                 # sinfo to see names; often gpu/a100/short/debug
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1             # ONE task = one torchrun launcher per node
#SBATCH --gres=gpu:a100:4               # older syntax, still the most portable
##SBATCH --gpus-per-node=4              # newer equivalent (Slurm >= 19.05); do not set both
#SBATCH --cpus-per-task=32              # 32 CPUs for THAT one task
#SBATCH --mem=240G                      # HOST RAM. Nothing to do with GPU memory.
#SBATCH --time=24:00:00                 # D-HH:MM:SS or HH:MM:SS
#SBATCH --output=logs/%x-%j.out         # %x=job name, %j=job id, %A_%a for arrays
#SBATCH --error=logs/%x-%j.err          # omit to merge stderr into --output
#SBATCH --open-mode=append              # CRITICAL if you requeue: default truncates the log
#SBATCH --requeue                       # allow requeue on preemption/node failure
#SBATCH --signal=B:USR1@120             # SIGUSR1 to the batch shell 120s before the wall clock

logs/ must exist before you submit. SLURM does not create it; the job starts, fails to open the output file, and dies with no log telling you why.

The trap that silently halves throughput: --cpus-per-task is per task; you get --ntasks × --cpus-per-task.

Two more CPU landmines:

1. Slurm 22.05+ stopped propagating --cpus-per-task from sbatch to srun. Your srun step silently gets 1 CPU per task. Always do this in the script:

   export SRUN_CPUS_PER_TASK="${SLURM_CPUS_PER_TASK:-1}"   # or pass --cpus-per-task to srun
   

2. os.cpu_count() and multiprocessing.cpu_count() report the whole physical node, not your cgroup. On a 128-core node with --cpus-per-task=8, num_workers=os.cpu_count()//2 spawns 64 workers into 8 cores. Use the affinity mask:

   import os
   n_cpu = len(os.sched_getaffinity(0))          # Linux: respects the cgroup / cpuset
   num_workers = max(1, n_cpu // int(os.environ.get("SLURM_GPUS_ON_NODE", 1)))
   

--mem is host RAM enforced by cgroups; exceed it and the kernel OOM killer SIGKILLs you. GPU memory is governed only by --gres=gpu:N — no SLURM flag limits or reserves VRAM.

Single-node multi-GPU script

Directives as above (--nodes=1 --ntasks-per-node=1 --gres=gpu:4 --cpus-per-task=32 --mem=200G), then the body:

set -euo pipefail                 # fail loudly; without this a bad `module load` is ignored

# --- environment (never rely on your interactive shell's env; see Environment section) ---
module purge
module load cuda/12.1 cudnn/8.9
source "$(conda info --base)/etc/profile.d/conda.sh"    # required: `conda activate` is a shell fn
conda activate dl

export SRUN_CPUS_PER_TASK="$SLURM_CPUS_PER_TASK"
export OMP_NUM_THREADS=$(( SLURM_CPUS_PER_TASK / SLURM_GPUS_ON_NODE ))   # else each rank grabs all cores
export PYTHONUNBUFFERED=1         # otherwise your log stays empty until the job dies
export TOKENIZERS_PARALLELISM=false

# --- stage data to node-local NVMe (see Data section) ---
LOCAL="${SLURM_TMPDIR:-${TMPDIR:-/tmp/$SLURM_JOB_ID}}"
mkdir -p "$LOCAL/data"
time tar -xf "/scratch/$USER/datasets/brats-shards.tar" -C "$LOCAL/data"

CKPT_DIR="/scratch/$USER/runs/$SLURM_JOB_NAME"; mkdir -p "$CKPT_DIR"

srun torchrun --standalone --nnodes=1 --nproc_per_node="$SLURM_GPUS_ON_NODE" \
  train.py --data "$LOCAL/data" --ckpt-dir "$CKPT_DIR" --num-workers 8 --resume auto &

wait      # required for the SIGUSR1 trap to fire; see Checkpointing section

--standalone sets up single-node rendezvous on a random free port — use it whenever --nodes=1, and you never touch MASTER_ADDR/MASTER_PORT.

Multi-node distributed training

Pick one launcher and do not mix them.

Static rendezvous (--node_rank) is deterministic and fine for fixed allocations; --rdzv_backend=c10d is needed for elastic runs but still requires node 0 reachable at the endpoint, so it does not remove the MASTER_ADDR work.

#!/bin/bash
#SBATCH --job-name=vitl-pretrain
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1          # ONE torchrun per node
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=32
#SBATCH --mem=400G
#SBATCH --time=2-00:00:00
#SBATCH --output=logs/%x-%j.out
#SBATCH --open-mode=append
#SBATCH --requeue
#SBATCH --signal=B:USR1@180
set -euo pipefail

module purge && module load cuda/12.1
source "$(conda info --base)/etc/profile.d/conda.sh" && conda activate dl

# --- rendezvous ---
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
# Port derived from the job id: unique per job, stable across requeue, above 1024 and below
# the 32768-60999 ephemeral range so it can't collide with a socket the OS picked.
MASTER_PORT=$(( 20000 + SLURM_JOB_ID % 10000 ))
export MASTER_ADDR MASTER_PORT
echo "rdzv $MASTER_ADDR:$MASTER_PORT  nodes=$SLURM_NNODES  gpus/node=$SLURM_GPUS_ON_NODE"

# --- interconnect: pick the fast NIC, or NCCL may fall back to 1GbE and you lose 20x ---
export NCCL_SOCKET_IFNAME=^lo,docker0,virbr0    # or the explicit device, e.g. ib0
export NCCL_DEBUG=WARN                          # INFO when debugging a hang, it is very verbose
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1        # torch>=2.2 (was NCCL_ASYNC_ERROR_HANDLING)
export SRUN_CPUS_PER_TASK="$SLURM_CPUS_PER_TASK"
export OMP_NUM_THREADS=$(( SLURM_CPUS_PER_TASK / SLURM_GPUS_ON_NODE ))
export PYTHONUNBUFFERED=1

# --- stage data ONCE PER NODE, before any torch process starts (see NCCL timeout note) ---
export LOCAL="${SLURM_TMPDIR:-/tmp/$SLURM_JOB_ID}"
export CKPT_DIR="/scratch/$USER/runs/$SLURM_JOB_NAME"; mkdir -p "$CKPT_DIR"
srun --ntasks-per-node=1 bash -c \
  'mkdir -p "$LOCAL/data" && tar -xf '"/scratch/$USER"'/datasets/shards.tar -C "$LOCAL/data"'

# Single quotes are load-bearing: SLURM_NODEID is set PER TASK by srun inside the step, and
# exported vars (MASTER_ADDR/PORT, LOCAL, CKPT_DIR) are propagated into it by srun.
srun --ntasks-per-node=1 --cpus-per-task="$SLURM_CPUS_PER_TASK" bash -c '
  torchrun --nnodes="$SLURM_NNODES" --nproc_per_node="$SLURM_GPUS_ON_NODE" \
           --node_rank="$SLURM_NODEID" \
           --master_addr="$MASTER_ADDR" --master_port="$MASTER_PORT" \
           train.py --data "$LOCAL/data" --ckpt-dir "$CKPT_DIR" --resume auto' &

wait

If you let the outer shell expand $SLURM_NODEID (double quotes, or a heredoc without quoting), it is baked in once by the batch shell — which runs on the first allocated node, where SLURM_NODEID is 0 (or unset) — so all four nodes launch with node_rank=0. The job then hangs at rendezvous until the store/NCCL timeout, with no error message pointing at it. Same trap for SLURM_PROCID/SLURM_LOCALID: per-task, meaningful only inside an srun step.

In train.py:

import os, torch, datetime, torch.distributed as dist

local_rank = int(os.environ["LOCAL_RANK"])       # set by torchrun
torch.cuda.set_device(local_rank)                # BEFORE init_process_group, always
dist.init_process_group("nccl", timeout=datetime.timedelta(minutes=60))  # default ~10 min
model = torch.nn.parallel.DistributedDataParallel(model.cuda(), device_ids=[local_rank])

NCCL's collective watchdog aborts the job if any rank is more than the timeout behind. The classic trigger is slow one-time work (untarring data, building an index, downloading weights) on rank 0 while the others wait at a barrier. Do that before init_process_group, or in the sbatch script via srun --ntasks-per-node=1, or raise timeout.

Debugging a multi-node hang, in order: (1) NCCL_DEBUG=INFO, check every rank printed a ring topology; (2) TORCH_DISTRIBUTED_DEBUG=DETAIL to catch mismatched collective shapes/order; (3) confirm all ranks have the same number of batches — use drop_last=True with DistributedSampler or a short rank desyncs the whole job.

Job arrays for sweeps

#SBATCH --array=0-23%4        # 24 tasks, at most 4 running concurrently
#SBATCH --output=logs/%x-%A_%a.out    # %A = array job id, %a = task index
#SBATCH --gres=gpu:1                  # resources are PER TASK, not for the whole array

The %4 throttle caps how much of the partition you occupy, so the scheduler still backfills your other work and you do not trip QOSMaxJobsPerUser. Without it, 23 tasks pend behind your own first task's priority drop.

--array=0-99:5 steps by 5. --array=3,7,11 reruns just the failed ones (find them with sacct -X -j <arrayjobid> -s FAILED,TIMEOUT -o JobID). The hard cap is MaxArraySize (often 1001, and it caps the index, not the count) — check scontrol show config.

# configs.txt, one run per line:  --lr 1e-4 --wd 0.05 --model vit_base
CFG=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" configs.txt)   # sed is 1-indexed, array is 0-indexed
[ -z "$CFG" ] && { echo "no config for task $SLURM_ARRAY_TASK_ID"; exit 1; }

OUT=/scratch/$USER/sweeps/$SLURM_JOB_NAME/${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}
mkdir -p "$OUT"
echo "$CFG" > "$OUT/config.txt"          # so you can reconstruct the run months later
srun python train.py $CFG --out-dir "$OUT" --seed "$SLURM_ARRAY_TASK_ID"

Two things that ruin sweeps: (a) every task writing to the same --out-dir or the same wandb run name, so results silently interleave; (b) $SLURM_ARRAY_TASK_ID unset when you resubmit without --array, making sed -n "1p" rerun config 0. The [ -z "$CFG" ] guard plus set -u catches both. sbatch --array=3,7,11 sweep.sh on the command line overrides the directive — handy for rerunning a subset without editing the file.

Checkpointing, atomicity, and requeue

Assume the job will be killed mid-write — time limit, preemption, node failure, full disk.

1. Checkpoint on wall clock (every 20-30 min), not only per epoch — one epoch on a large dataset can exceed the partition's time limit.
2. Write to a temp file on the same filesystem, fsync, then os.replace (atomic on POSIX). A kill mid-torch.save otherwise leaves a truncated file that fails to load and has already destroyed your last good one.
3. Save optimizer, LR scheduler, AMP scaler, epoch and global step, and RNG state. Restoring only model weights restarts the LR schedule and reuses the same data order — you get a visible discontinuity in the loss curve and a run that is not the one you claim in the paper.
4. Resume automatically at startup. A resume you have to remember to type is one you will forget at 3am.

import os, random, tempfile, numpy as np, torch, torch.distributed as dist

def save_checkpoint(path, model, optimizer, scheduler, scaler, epoch, step):
    if dist.is_initialized() and dist.get_rank() != 0:
        dist.barrier()                    # non-zero ranks wait; keeps ranks in lockstep
        return
    state = {
        "model": (model.module if hasattr(model, "module") else model).state_dict(),
        "optimizer": optimizer.state_dict(),
        "scheduler": scheduler.state_dict(),
        "scaler": scaler.state_dict(),        # AMP loss scale; dropping it spikes loss on resume
        "epoch": epoch, "step": step,
        "rng": {
            "python": random.getstate(),
            "numpy": np.random.get_state(),
            "torch": torch.get_rng_state(),                 # CPU generator
            "cuda": torch.cuda.get_rng_state_all(),         # list, one per visible device
        },
    }
    d = os.path.dirname(path) or "."
    os.makedirs(d, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=d, suffix=".tmp")        # same dir => same filesystem
    try:
        with os.fdopen(fd, "wb") as f:
            torch.save(state, f)
            f.flush()
            os.fsync(f.fileno())                            # force to disk before the rename
        os.replace(tmp, path)                               # atomic
        link_tmp = path + ".link.tmp"                        # point "latest" at it, atomically
        if os.path.lexists(link_tmp): os.remove(link_tmp)
        os.symlink(os.path.basename(path), link_tmp)
        os.replace(link_tmp, os.path.join(d, "latest.pt"))
    except BaseException:
        if os.path.exists(tmp): os.remove(tmp)
        raise
    if dist.is_initialized():
        dist.barrier()

def load_checkpoint(path, model, optimizer, scheduler, scaler, device):
    ck = torch.load(path, map_location=device, weights_only=False)  # your own file: trusted
    (model.module if hasattr(model, "module") else model).load_state_dict(ck["model"])
    optimizer.load_state_dict(ck["optimizer"]); scheduler.load_state_dict(ck["scheduler"])
    scaler.load_state_dict(ck["scaler"])
    random.setstate(ck["rng"]["python"]); np.random.set_state(ck["rng"]["numpy"])
    torch.set_rng_state(ck["rng"]["torch"].cpu())            # must be a CPU ByteTensor
    torch.cuda.set_rng_state_all([s.cpu() for s in ck["rng"]["cuda"]])
    return ck["epoch"], ck["step"]

Notes that bite: weights_only=True is the safe default in torch 2.6+ but rejects your own checkpoint, because the RNG dict holds non-tensor Python objects — use weights_only=False for files you wrote, never for downloaded ones. torch.cuda.get_rng_state_all() length follows CUDA_VISIBLE_DEVICES, so a 4-GPU checkpoint will not restore onto 2 GPUs; guard the length. Keep the last 2-3 checkpoints, not one: atomic rename protects you from truncation, not from having saved a NaN model.

This rank-0-only pattern is correct for DDP only. Under FSDP or DeepSpeed ZeRO-2/3 each rank holds a shard of the params/optimizer state, so a rank-0 state_dict() is either wrong or a collective that every rank must enter. Call save_checkpoint on all ranks and let the framework do the gathering: FSDP via torch.distributed.checkpoint.save (one file per rank, resharded on load) or FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, FullStateDictConfig(offload_to_cpu=True, rank0_only=True)) before the rank-0 branch; DeepSpeed via engine.save_checkpoint(dir, tag) on every rank. Full consolidated state for a 7B model is ~80 GB with optimizer state — sharded saves are also 10-50x faster to the shared filesystem, which is what makes the @120 signal window survivable.

Preemption / time-limit handler

#SBATCH --signal=B:USR1@120     # B: = signal the BATCH SHELL, not the job steps
#SBATCH --requeue

Without B:, SIGUSR1 goes to the srun-launched processes and your bash trap never runs. @120 is seconds before the wall clock; make it comfortably longer than one checkpoint write — a 7B optimizer state to a shared filesystem can take minutes. Measure it, do not guess.

handler() {
  echo "$(date -Is) caught USR1, requeueing job $SLURM_JOB_ID"
  # Ask the training process to checkpoint. Either forward the signal to python
  # (which installs its own signal.SIGUSR1 handler and saves), or drop a sentinel
  # file that the training loop polls once per step.
  touch "$CKPT_DIR/PLEASE_CHECKPOINT"
  sleep 90                      # give it time to finish the atomic write
  scontrol requeue "$SLURM_JOB_ID"
  exit 0
}
trap handler USR1

srun ... train.py ... &         # MUST be backgrounded
wait                            # bash only runs traps between foreground commands;
                                # `wait` is interruptible, a foreground srun is not.

Requeued jobs keep the same job id, so --output=logs/%x-%j.out reopens the same file — this is exactly why --open-mode=append matters; the default truncate erases the first run's log. SLURM_RESTART_COUNT tells you which attempt you are on; log it and bail out above ~10 to avoid an infinite requeue loop against a genuinely broken node.

The sentinel file is more robust than forwarding signals through srun → torchrun → N python children, where propagation is version-dependent. Poll it in the training loop:

if step % 50 == 0 and os.path.exists(os.path.join(ckpt_dir, "PLEASE_CHECKPOINT")):
    save_checkpoint(...); dist.barrier(); sys.exit(0)

Auto-resume at startup: load ckpt_dir/latest.pt if it exists, else start fresh. That single branch makes requeue transparent.

Diagnosing failures

Out of memory — decide which memory first

They are unrelated systems and the fix is different:

Reading MaxRSS correctly: it is recorded on the step rows (12345.batch, 12345.0), not the job row, so sacct -X -j 12345 -o MaxRSS shows blank — drop -X. It is also sampled every JobAcctGatherFrequency seconds (commonly 30), so a fast spike is missed entirely: a job that OOM-kills while MaxRSS looks fine is normal, not a contradiction.

Host-RAM math people get wrong: each DataLoader worker is a fork with its own copy-on-write memory, and decoded batches sit in pinned host memory. num_workers=16, prefetch_factor=4, batch 32 of 512×512×3 float32 ≈ 16×4×32×3 MB ≈ 6 GB of buffers alone.

Job pending forever

squeue -u $USER -o "%.10i %.9P %.2t %R" — the last column is the reason.

A request no node can ever satisfy (e.g. --gres=gpu:8 on a partition of 4-GPU nodes) pends forever rather than erroring. sinfo -p <part> --Node -o "%N %G" shows what actually exists; sbatch --test-only catches many of these at submit time.

Time limit hit / node failure

TIMEOUT means the log just stops; sacct -o Elapsed,Timelimit confirms. Only a disaster if you were not checkpointing. Structurally, prefer many short requeueing jobs over one long one — short jobs backfill and start far sooner than a 48h request.

NODE_FAIL (or srun: error: ... Killed by node failure) is retried automatically with --requeue. If the same node kills you repeatedly, get it from sacct -j ID -o NodeList, resubmit with --exclude=gpu07, and report it.

"Works interactively, fails in batch"

Almost always environment, and almost always one of these:

• ~/.bashrc is not sourced. #!/bin/bash in a batch script is a non-interactive, non-login shell. Your conda init block and PATH edits do not run. Fix: explicitly source "$(conda info --base)/etc/profile.d/conda.sh" then conda activate. Do not use #!/bin/bash -l as the fix — it works, but it drags in whatever is in your login files, which is the next bullet.
• --export=ALL is the default, so your login shell's PYTHONPATH, LD_LIBRARY_PATH, CUDA_VISIBLE_DEVICES and half-activated conda env leak into the job. It works for you and breaks for your collaborator. For reproducibility: #SBATCH --export=NONE and set up the environment explicitly in the script. (SLURM_* variables are still provided.)
• Different modules on login vs compute nodes, or a login node with a newer CPU. Always module purge first, then load pinned versions (module load cuda/12.1, not cuda).
• No set -euo pipefail, so a failed module load or conda activate prints a warning and the job runs against the system Python, producing a confusing ImportError 40 lines later.
• Relative paths. The job starts in $SLURM_SUBMIT_DIR, which is where you submitted from, not where the script lives. Use absolute paths or cd "$SLURM_SUBMIT_DIR".
• CUDA_VISIBLE_DEVICES. SLURM already restricts you to your allocated GPUs; if you also export it yourself you will index into the wrong set. Never set it in an sbatch script.

Reproduce the batch environment on purpose: srun --export=NONE --pty bash --noprofile --norc, then run your setup lines by hand.

Containers (Singularity/Apptainer) remove this whole class of problem and are the right answer for anything you will publish:

srun singularity exec --nv \
  --bind "$LOCAL/data":/data,/scratch/$USER/runs:/runs \
  /scratch/$USER/images/torch24.sif \
  python /workspace/train.py --data /data --ckpt-dir /runs

--nv is what exposes the NVIDIA driver — forget it and you get "no CUDA-capable device". Bind what you need explicitly rather than assuming $HOME and /scratch are visible. Build the .sif where you have root (or via a remote builder); you generally cannot build one on the cluster.

Compute nodes usually have no outbound internet

Login nodes reach the internet; compute nodes commonly do not. The failure is not an immediate error — HTTP calls hang until a long timeout, so the job burns its first 10 minutes of GPU time and then dies (or worse, W&B silently drops your metrics). Anything that downloads must be done on the login node first, into a shared cache, and then pinned offline in the job:

# ONCE, on the login node (never in the job):
export HF_HOME=/scratch/$USER/hf                  # replaces the old TRANSFORMERS_CACHE
huggingface-cli download facebook/dinov2-large    # weights, tokenizer, config into HF_HOME
python -c "import timm; timm.create_model('vit_large_patch14_dinov2', pretrained=True)"

# in the sbatch script:
export HF_HOME=/scratch/$USER/hf
export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1     # fail fast instead of hanging on a 30s retry
export WANDB_MODE=offline                         # writes to ./wandb/offline-run-*
export WANDB_DIR="$CKPT_DIR"                      # keep runs with the checkpoints, not in $HOME
export TORCH_HOME=/scratch/$USER/torch            # torchvision / torch.hub weights

Then wandb sync "$CKPT_DIR"/wandb/offline-run-* from the login node afterwards. For a sweep, set WANDB_RUN_GROUP="$SLURM_ARRAY_JOB_ID" and WANDB_NAME="task-$SLURM_ARRAY_TASK_ID" so the 24 offline runs land as one comparable group instead of 24 identically named runs.

Data: the shared filesystem is the bottleneck

Lustre/GPFS/NFS are optimized for large sequential I/O from few clients. A dataset of 2 million loose PNGs hammers the metadata server, and every other user on the cluster notices. Symptoms: GPU utilization sawtoothing between 0 and 90%, ls taking 30 seconds, epoch times that vary wildly and get worse when the cluster is busy.

Order of operations at job start:

1. Keep the dataset on shared storage as a small number of large archives — WebDataset .tar shards (~0.5-2 GB each), or LMDB, or HDF5. 500 shards, not 2M files.
2. Copy/extract to node-local scratch once per node: srun --ntasks-per-node=1 tar -xf ... -C "$SLURM_TMPDIR". Doing it per-rank means 4 processes untarring the same archive onto each other.
3. Train from node-local scratch. Write checkpoints to shared /scratch (node-local is wiped when the job ends — anything left there is gone, including your last checkpoint).

Which variable: $SLURM_TMPDIR exists on Alliance-style clusters; elsewhere it may be $TMPDIR, /local/$SLURM_JOB_ID, or nothing at all. Use LOCAL="${SLURM_TMPDIR:-${TMPDIR:-/tmp/$SLURM_JOB_ID}}" and check df -h "$LOCAL" in a debug job before assuming you have 2 TB of NVMe.

Also: num_workers>0 + persistent_workers=True + pin_memory=True is the right default once data is local; on a slow shared FS more workers just multiplies metadata pressure.

Etiquette and not wasting the allocation

• Debug first. Every cluster has a short/debug partition or QOS with near-instant start. Run the exact script with --time=00:15:00 and a --max-steps 20 flag before submitting 48h.
• Request honestly. Over-requesting delays you: the scheduler backfills small, short jobs into gaps, so a 4h/1-GPU job often starts hours before a 24h/4-GPU one. Right-size from seff/sacct of a previous run rather than guessing.
• Check seff <jobid> after every substantial run.

  CPU Efficiency: 8.2% of 32-00:00:00 core-walltime     <- asked for 32 cores, used ~2.6
  Memory Efficiency: 11.4% of 200.00 GB                 <- drop --mem to 32G and start sooner
  

  seff does not report GPU utilization, the number that matters most. Log it yourself on the first run of a new config:

  nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv -l 60 \
      > "$CKPT_DIR/gpustats-$SLURM_JOB_ID.csv" &
  NVSMI_PID=$!;  trap 'kill $NVSMI_PID 2>/dev/null' EXIT
  

  Sustained GPU util below ~80% on a data-heavy vision job means you are input-bound: check num_workers, whether data is on node-local scratch, and whether decode/augmentation is the hot path.
• Chain long runs instead of asking for a week: JID=$(sbatch --parsable train.sh); sbatch --dependency=afterany:$JID train.sh — with auto-resume from latest.pt this gets 5 days out of a 24h limit.
• Never run training, tarring, or large rsync on the login node.

Related

The qzcli skill covers the Qizhi platform's own submission tooling; use it there. This skill is for vanilla SLURM (sbatch/srun/sacct) on a generic HPC cluster.