Palmetto Slurm Workflow
Use This Workflow
When writing or modifying long-running sbatch scripts, also apply
palmetto-slurm-safe-guard for safety checks, save/recovery, and low-noise monitoring.
When the job is a training job, also apply exact-training-resume-guard so checkpoints support exact resume instead of weight-only restart.
- Classify request type first.
- Onboarding/login/account/docs links
- Slurm resource request and job submission
- Environment setup (module + conda + pip)
- Data/code sync between laptop and cluster
- PyCharm or Jupyter remote workflow
- Enforce cluster-safe execution rules.
- Run heavy compute only on Slurm-allocated compute nodes.
- Keep login node for editing,
scp/rsync, and job submission/monitoring.
- Keep
/home for code, configs, notebooks, manifests, and standard lightweight experiment results.
- Prefer
$SCRATCH for large datasets, caches, model weights, checkpoints, container images, raw predictions, and any large file or large directory.
- Keep code-generated logs, manifests, compact metrics, and small standard experiment outputs inside the project directory by default unless the user explicitly requests another location.
- If outputs are large or append-heavy, write them to scratch and keep only summaries, final tables, or small derived results in the project directory.
- Choose execution mode.
- For quick debug: use
salloc or srun --pty bash.
- For training/long runs: use
sbatch and tail logs.
- For long-running training, require full-state checkpoints and latest-complete-checkpoint resume behavior.
- Emit actionable commands, not generic advice.
- Provide exact command blocks with placeholder variables.
- Include one minimal example command and one production command.
- Validate common failure points.
- Wrong partition/account/reservation
- Missing
module load/conda activate
- Running training on login node
- GPU count mismatch between Slurm allocation and
torchrun
- Training script saves model weights only and cannot exact-resume after timeout/preemption
- Home quota or scratch path issues
- In Codex/agent sessions,
$USER may be coder; do not build /scratch/$USER/... or /home/$USER/... paths for submitted jobs unless that value is verified. Prefer explicit absolute user paths.
- Logs or run artifacts accidentally written outside the repository when the user expects project-local reproducibility records
- Large run artifacts accidentally written under
/home when they should be on scratch
Required References
- Use palmetto-core.md for Slurm/login/sync patterns.
- Use palmetto-ide-jupyter.md for PyCharm/Jupyter setup and SSH tunneling.
Load only the reference file needed for the user request.
1---2name: palmetto-slurm-workflow3description: Run Clemson Palmetto HPC workflows with Slurm and remote development tooling. Use when requests mention Palmetto, Slurm (`sbatch`/`salloc`/`srun`), login-node vs compute-node usage, conda/module environment setup, file sync between laptop and cluster, PyCharm remote interpreter setup, or Jupyter SSH tunneling.4metadata:5 source: https://skillmd.com/skills/kwongfuk/palmetto-slurm-workflow6---78# Palmetto Slurm Workflow910## Use This Workflow1112When writing or modifying long-running `sbatch` scripts, also apply13`palmetto-slurm-safe-guard` for safety checks, save/recovery, and low-noise monitoring.14When the job is a training job, also apply `exact-training-resume-guard` so checkpoints support exact resume instead of weight-only restart.15161. Classify request type first.17- Onboarding/login/account/docs links18- Slurm resource request and job submission19- Environment setup (module + conda + pip)20- Data/code sync between laptop and cluster21- PyCharm or Jupyter remote workflow22232. Enforce cluster-safe execution rules.24- Run heavy compute only on Slurm-allocated compute nodes.25- Keep login node for editing, `scp/rsync`, and job submission/monitoring.26- Keep `/home` for code, configs, notebooks, manifests, and standard lightweight experiment results.27- Prefer `$SCRATCH` for large datasets, caches, model weights, checkpoints, container images, raw predictions, and any large file or large directory.28- Keep code-generated logs, manifests, compact metrics, and small standard experiment outputs inside the project directory by default unless the user explicitly requests another location.29- If outputs are large or append-heavy, write them to scratch and keep only summaries, final tables, or small derived results in the project directory.30313. Choose execution mode.32- For quick debug: use `salloc` or `srun --pty bash`.33- For training/long runs: use `sbatch` and tail logs.34- For long-running training, require full-state checkpoints and latest-complete-checkpoint resume behavior.35364. Emit actionable commands, not generic advice.37- Provide exact command blocks with placeholder variables.38- Include one minimal example command and one production command.39405. Validate common failure points.41- Wrong partition/account/reservation42- Missing `module load`/`conda activate`43- Running training on login node44- GPU count mismatch between Slurm allocation and `torchrun`45- Training script saves model weights only and cannot exact-resume after timeout/preemption46- Home quota or scratch path issues47- In Codex/agent sessions, `$USER` may be `coder`; do not build `/scratch/$USER/...` or `/home/$USER/...` paths for submitted jobs unless that value is verified. Prefer explicit absolute user paths.48- Logs or run artifacts accidentally written outside the repository when the user expects project-local reproducibility records49- Large run artifacts accidentally written under `/home` when they should be on scratch5051## Required References5253- Use [palmetto-core.md](references/palmetto-core.md) for Slurm/login/sync patterns.54- Use [palmetto-ide-jupyter.md](references/palmetto-ide-jupyter.md) for PyCharm/Jupyter setup and SSH tunneling.5556Load only the reference file needed for the user request.