Palmetto Slurm Workflow
Use This Workflow
When writing or modifying long-running sbatch scripts, also apply
palmetto-slurm-safe-guard for safety checks, save/recovery, and low-noise monitoring.
When the job is a training job, also apply exact-training-resume-guard so checkpoints support exact resume instead of weight-only restart.
- Classify request type first.
- Onboarding/login/account/docs links
- Slurm resource request and job submission
- Environment setup (module + conda + pip)
- Data/code sync between laptop and cluster
- PyCharm or Jupyter remote workflow
- Enforce cluster-safe execution rules.
- Run heavy compute only on Slurm-allocated compute nodes.
- Keep login node for editing,
scp/rsync, and job submission/monitoring.
- Keep
/home for code, configs, notebooks, manifests, and standard lightweight experiment results.
- Prefer
$SCRATCH for large datasets, caches, model weights, checkpoints, container images, raw predictions, and any large file or large directory.
- Keep code-generated logs, manifests, compact metrics, and small standard experiment outputs inside the project directory by default unless the user explicitly requests another location.
- If outputs are large or append-heavy, write them to scratch and keep only summaries, final tables, or small derived results in the project directory.
- Choose execution mode.
- For quick debug: use
salloc or srun --pty bash.
- For training/long runs: use
sbatch and tail logs.
- For long-running training, require full-state checkpoints and latest-complete-checkpoint resume behavior.
- Emit actionable commands, not generic advice.
- Provide exact command blocks with placeholder variables.
- Include one minimal example command and one production command.
- Validate common failure points.
- Wrong partition/account/reservation
- Missing
module load/conda activate
- Running training on login node
- GPU count mismatch between Slurm allocation and
torchrun
- Training script saves model weights only and cannot exact-resume after timeout/preemption
- Home quota or scratch path issues
- In Codex/agent sessions,
$USER may be coder; do not build /scratch/$USER/... or /home/$USER/... paths for submitted jobs unless that value is verified. Prefer explicit absolute user paths.
- Logs or run artifacts accidentally written outside the repository when the user expects project-local reproducibility records
- Large run artifacts accidentally written under
/home when they should be on scratch
Required References
- Use palmetto-core.md for Slurm/login/sync patterns.
- Use palmetto-ide-jupyter.md for PyCharm/Jupyter setup and SSH tunneling.
Load only the reference file needed for the user request.
1---2name: palmetto-slurm-workflow3description: Run Clemson Palmetto HPC workflows with Slurm and remote development tooling. Use when requests mention Palmetto, Slurm (`sbatch`/`salloc`/`srun`), login-node vs compute-node usage, conda/module environment setup, file sync between laptop and cluster, PyCharm remote interpreter setup, or Jupyter SSH tunneling.4---56# Palmetto Slurm Workflow78## Use This Workflow910When writing or modifying long-running `sbatch` scripts, also apply11`palmetto-slurm-safe-guard` for safety checks, save/recovery, and low-noise monitoring.12When the job is a training job, also apply `exact-training-resume-guard` so checkpoints support exact resume instead of weight-only restart.13141. Classify request type first.15- Onboarding/login/account/docs links16- Slurm resource request and job submission17- Environment setup (module + conda + pip)18- Data/code sync between laptop and cluster19- PyCharm or Jupyter remote workflow20212. Enforce cluster-safe execution rules.22- Run heavy compute only on Slurm-allocated compute nodes.23- Keep login node for editing, `scp/rsync`, and job submission/monitoring.24- Keep `/home` for code, configs, notebooks, manifests, and standard lightweight experiment results.25- Prefer `$SCRATCH` for large datasets, caches, model weights, checkpoints, container images, raw predictions, and any large file or large directory.26- Keep code-generated logs, manifests, compact metrics, and small standard experiment outputs inside the project directory by default unless the user explicitly requests another location.27- If outputs are large or append-heavy, write them to scratch and keep only summaries, final tables, or small derived results in the project directory.28293. Choose execution mode.30- For quick debug: use `salloc` or `srun --pty bash`.31- For training/long runs: use `sbatch` and tail logs.32- For long-running training, require full-state checkpoints and latest-complete-checkpoint resume behavior.33344. Emit actionable commands, not generic advice.35- Provide exact command blocks with placeholder variables.36- Include one minimal example command and one production command.37385. Validate common failure points.39- Wrong partition/account/reservation40- Missing `module load`/`conda activate`41- Running training on login node42- GPU count mismatch between Slurm allocation and `torchrun`43- Training script saves model weights only and cannot exact-resume after timeout/preemption44- Home quota or scratch path issues45- In Codex/agent sessions, `$USER` may be `coder`; do not build `/scratch/$USER/...` or `/home/$USER/...` paths for submitted jobs unless that value is verified. Prefer explicit absolute user paths.46- Logs or run artifacts accidentally written outside the repository when the user expects project-local reproducibility records47- Large run artifacts accidentally written under `/home` when they should be on scratch4849## Required References5051- Use [palmetto-core.md](references/palmetto-core.md) for Slurm/login/sync patterns.52- Use [palmetto-ide-jupyter.md](references/palmetto-ide-jupyter.md) for PyCharm/Jupyter setup and SSH tunneling.5354Load only the reference file needed for the user request.
KwongFuk