stampede3-debug — diagnose Stampede3 Slurm jobs
Triage workflow
Ask for one of: a JobID, a stderr/stdout file, or the submitted script + symptom. Then run a fixed sequence:
1. Pending? Find out why.
squeue -j <jobid> -o "%i %P %T %r %S %L"
%r (Reason) is the answer. Common values:
Priority / Resources — normal, just waiting. Estimated start: --start flag.QOSMaxJobsPerUserLimit — too many of your jobs queued; check qlimits.AssocGrpCPUMinutesLimit — allocation is out of SUs. Confirm with /usr/local/etc/taccinfo.PartitionNodeLimit — you asked for more nodes than the queue allows (see the queue table in the stampede3-submit skill).ReqNodeNotAvail — usually a maintenance reservation; check sinfo -R and the TACC user news.Dependency — waiting on another job (scontrol show job <id> to see which).
2. Completed/failed? Pull sacct.
sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,Timelimit,MaxRSS,ReqMem,NodeList,Partition,Start,End
Decode State + ExitCode:
COMPLETED with 0:0 — clean exit.TIMEOUT — hit -t wall. Resubmit with longer time or checkpoint.OUT_OF_MEMORY / OOM in dmesg — see §4.CANCELLED by <uid> — user or admin scancel.NODE_FAIL — hardware. Resubmit; report to TACC if repeated on same node.FAILED 1:0 — program returned non-zero. Read stderr.FAILED 0:9 / 0:15 — killed by SIGKILL/SIGTERM (usually OOM or wall).FAILED 0:53 — often segfault from MPI launch issues.
3. Read the logs (in order).
ls -la slurm-<jobid>*.out
tail -100 slurm-<jobid>.out
Grep for these red flags:
slurmstepd: error: Exceeded job memory limit → OOM.srun: error or MPI_ABORT → MPI launch / rank failure.Illegal instruction → AVX-512 binary running on wrong arch (e.g., SPR build on SKX).Disk quota exceeded → $HOME or $WORK is full. Check /usr/local/etc/tacc_quota.No such file or directory referencing $SCRATCH/... → purged. Files older than 10-day atime are removed.command not found: ibrun → submitted from a context without TACC modules; add module reset to script.Lmod has detected the following error → module conflict; rebuild env with module reset && module load <fresh>.
4. Confirm OOM specifically.
sacct -j <jobid> --format=JobID,MaxRSS,MaxVMSize,ReqMem,NodeList
MaxRSS near node RAM (SKX 192 GB, ICX 256 GB, SPR 128 GB HBM, NVDIMM 4 TB, GPU nodes 1 TB) means OOM.- SPR trap: only 128 GB HBM/node. Codes that fit on SKX/ICX may OOM on SPR. Either fewer ranks per node, or move to
icx/nvdimm.
5. Wall time / efficiency.
sacct -j <jobid> --format=JobID,Elapsed,Timelimit,CPUTime,TotalCPU
seff <jobid> # if available
Elapsed ≈ Timelimit and state TIMEOUT → underestimated wall. Bump -t next run.CPUTime/TotalCPU ≪ Elapsed × ncpus → poor parallel scaling or wrong launcher (using mpirun instead of ibrun, or no MPI at all). Re-check the script.
6. Node-specific failure?
sacct -j <jobid> --format=JobID,NodeList,State
sinfo -n <node> -o "%n %T %E"
If the same node ID shows up across multiple failed jobs, mention it to the user and suggest a ticket to TACC.
Stampede3-specific gotchas (check these first)
| Symptom |
Likely cause |
Fix |
| MPI ranks not laid out across nodes |
Used mpirun/srun instead of ibrun |
Replace with ibrun |
Illegal instruction |
Binary built with -xSAPPHIRERAPIDS running on SKX/ICX |
Rebuild with -axCORE-AVX512,ICELAKE-SERVER,SAPPHIRERAPIDS or match build host to run queue |
Job script errors command not found for things in your .bashrc |
Slurm strips most environment; --export=... interferes |
Put module load lines in the script, drop --export |
Files vanished between jobs on $SCRATCH |
10-day atime purge |
Move long-lived data to $WORK |
Disk-full on $HOME mid-run |
Logs/checkpoints to $HOME (15 GB cap) |
Redirect output to $SCRATCH |
| H100 job sees 0 GPUs |
module load cuda missing or container started before module |
Add module load cuda to script |
| PVC job: SYCL device not found |
Missing module load oneapi |
Load oneapi (and verify with sycl-ls) |
MaxRSS looks low but job was killed |
Check per-node memory, not total — MaxRSS is per-step, not summed |
Re-check with --units=G and consider rank-per-node math |
Reporting back to the user
Give a structured diagnosis:
- What happened (one sentence, e.g., "Job 1234567 hit the 4-hour wall after 3h59m on
spr.")
- Evidence (the specific
sacct field or log line you keyed off)
- Fix (concrete change: bump
-t, switch partition, rebuild binary, etc.)
- Next step (offer to draft a corrected script via
stampede3-submit, or to dig deeper into a specific node/log)
Keep it tight. Don't dump the full sacct table unless asked — quote the field that mattered.
When the user is fishing ("my job is slow")
Without a JobID, ask for one. With one, look at:
seff <jobid> for CPU efficiencysacct ... CPUTime,TotalCPU,Elapsed ratioMaxRSS vs requested resources (over-asked = wasted SU; near-limit = OOM risk)ibrun vs other launcher in the script- Build flags vs run partition (AVX mismatch)
- I/O target — heavy I/O on
$HOME/$WORK is slow; should be $SCRATCH
Then suggest one targeted experiment, not a laundry list.
1---2name: stampede3-debug3description: Diagnose a failed, stuck, slow, or pending Slurm job on Stampede3 (TACC). Use when the user reports a JobID that crashed, exited early, ran out of memory/time, "is stuck in queue", produced unexpected output, or was killed. Walks through sacct, output logs, partition state, and Stampede3-specific failure modes (ibrun misuse, scratch purge, AVX mismatch, SPR HBM OOM). To write a new job script, use stampede3-submit.4---56# stampede3-debug — diagnose Stampede3 Slurm jobs78## Triage workflow910Ask for one of: a **JobID**, a **stderr/stdout file**, or the **submitted script + symptom**. Then run a fixed sequence:1112### 1. Pending? Find out why.13```bash14squeue -j <jobid> -o "%i %P %T %r %S %L"15```16`%r` (Reason) is the answer. Common values:17- `Priority` / `Resources` — normal, just waiting. Estimated start: `--start` flag.18- `QOSMaxJobsPerUserLimit` — too many of your jobs queued; check `qlimits`.19- `AssocGrpCPUMinutesLimit` — allocation is out of SUs. Confirm with `/usr/local/etc/taccinfo`.20- `PartitionNodeLimit` — you asked for more nodes than the queue allows (see the queue table in the `stampede3-submit` skill).21- `ReqNodeNotAvail` — usually a maintenance reservation; check `sinfo -R` and the TACC user news.22- `Dependency` — waiting on another job (`scontrol show job <id>` to see which).2324### 2. Completed/failed? Pull sacct.25```bash26sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,Timelimit,MaxRSS,ReqMem,NodeList,Partition,Start,End27```28Decode `State` + `ExitCode`:29- `COMPLETED` with `0:0` — clean exit.30- `TIMEOUT` — hit `-t` wall. Resubmit with longer time or checkpoint.31- `OUT_OF_MEMORY` / OOM in dmesg — see §4.32- `CANCELLED by <uid>` — user or admin scancel.33- `NODE_FAIL` — hardware. Resubmit; report to TACC if repeated on same node.34- `FAILED 1:0` — program returned non-zero. Read stderr.35- `FAILED 0:9` / `0:15` — killed by SIGKILL/SIGTERM (usually OOM or wall).36- `FAILED 0:53` — often segfault from MPI launch issues.3738### 3. Read the logs (in order).39```bash40ls -la slurm-<jobid>*.out41tail -100 slurm-<jobid>.out42```43Grep for these red flags:44- `slurmstepd: error: Exceeded job memory limit` → OOM.45- `srun: error` or `MPI_ABORT` → MPI launch / rank failure.46- `Illegal instruction` → AVX-512 binary running on wrong arch (e.g., SPR build on SKX).47- `Disk quota exceeded` → `$HOME` or `$WORK` is full. Check `/usr/local/etc/tacc_quota`.48- `No such file or directory` referencing `$SCRATCH/...` → purged. Files older than 10-day atime are removed.49- `command not found: ibrun` → submitted from a context without TACC modules; add `module reset` to script.50- `Lmod has detected the following error` → module conflict; rebuild env with `module reset && module load <fresh>`.5152### 4. Confirm OOM specifically.53```bash54sacct -j <jobid> --format=JobID,MaxRSS,MaxVMSize,ReqMem,NodeList55```56- `MaxRSS` near node RAM (SKX 192 GB, ICX 256 GB, SPR 128 GB HBM, NVDIMM 4 TB, GPU nodes 1 TB) means OOM.57- **SPR trap**: only 128 GB HBM/node. Codes that fit on SKX/ICX may OOM on SPR. Either fewer ranks per node, or move to `icx`/`nvdimm`.5859### 5. Wall time / efficiency.60```bash61sacct -j <jobid> --format=JobID,Elapsed,Timelimit,CPUTime,TotalCPU62seff <jobid> # if available63```64- `Elapsed` ≈ `Timelimit` and state `TIMEOUT` → underestimated wall. Bump `-t` next run.65- `CPUTime`/`TotalCPU` ≪ `Elapsed × ncpus` → poor parallel scaling or wrong launcher (using `mpirun` instead of `ibrun`, or no MPI at all). Re-check the script.6667### 6. Node-specific failure?68```bash69sacct -j <jobid> --format=JobID,NodeList,State70sinfo -n <node> -o "%n %T %E"71```72If the same node ID shows up across multiple failed jobs, mention it to the user and suggest a ticket to TACC.7374## Stampede3-specific gotchas (check these first)7576| Symptom | Likely cause | Fix |77|---------|--------------|-----|78| MPI ranks not laid out across nodes | Used `mpirun`/`srun` instead of `ibrun` | Replace with `ibrun` |79| `Illegal instruction` | Binary built with `-xSAPPHIRERAPIDS` running on SKX/ICX | Rebuild with `-axCORE-AVX512,ICELAKE-SERVER,SAPPHIRERAPIDS` or match build host to run queue |80| Job script errors `command not found` for things in your `.bashrc` | Slurm strips most environment; `--export=...` interferes | Put `module load` lines in the script, drop `--export` |81| Files vanished between jobs on `$SCRATCH` | 10-day atime purge | Move long-lived data to `$WORK` |82| Disk-full on `$HOME` mid-run | Logs/checkpoints to `$HOME` (15 GB cap) | Redirect output to `$SCRATCH` |83| H100 job sees 0 GPUs | `module load cuda` missing or container started before module | Add `module load cuda` to script |84| PVC job: SYCL device not found | Missing `module load oneapi` | Load oneapi (and verify with `sycl-ls`) |85| `MaxRSS` looks low but job was killed | Check per-node memory, not total — `MaxRSS` is per-step, not summed | Re-check with `--units=G` and consider rank-per-node math |8687## Reporting back to the user8889Give a structured diagnosis:901. **What happened** (one sentence, e.g., "Job 1234567 hit the 4-hour wall after 3h59m on `spr`.")912. **Evidence** (the specific `sacct` field or log line you keyed off)923. **Fix** (concrete change: bump `-t`, switch partition, rebuild binary, etc.)934. **Next step** (offer to draft a corrected script via `stampede3-submit`, or to dig deeper into a specific node/log)9495Keep it tight. Don't dump the full sacct table unless asked — quote the field that mattered.9697## When the user is fishing ("my job is slow")9899Without a JobID, ask for one. With one, look at:100- `seff <jobid>` for CPU efficiency101- `sacct ... CPUTime,TotalCPU,Elapsed` ratio102- `MaxRSS` vs requested resources (over-asked = wasted SU; near-limit = OOM risk)103- `ibrun` vs other launcher in the script104- Build flags vs run partition (AVX mismatch)105- I/O target — heavy I/O on `$HOME`/`$WORK` is slow; should be `$SCRATCH`106107Then suggest one targeted experiment, not a laundry list.
charliecpeterson