# Palmetto Slurm Workflow

> Run Clemson Palmetto HPC workflows with Slurm and remote development tooling. Use when requests mention Palmetto, Slurm (`sbatch`/`salloc`/`srun`), login-node vs compute-node usage, conda/module environment setup, file sync between laptop and cluster, PyCharm remote interpreter setup, or Jupyter SSH tunneling.

- Skill: `kwongfuk/palmetto-slurm-workflow` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add kwongfuk/palmetto-slurm-workflow`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/palmetto-slurm-workflow/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/palmetto-slurm-workflow

---


# Palmetto Slurm Workflow

## Use This Workflow

When writing or modifying long-running `sbatch` scripts, also apply
`palmetto-slurm-safe-guard` for safety checks, save/recovery, and low-noise monitoring.
When the job is a training job, also apply `exact-training-resume-guard` so checkpoints support exact resume instead of weight-only restart.

1. Classify request type first.
- Onboarding/login/account/docs links
- Slurm resource request and job submission
- Environment setup (module + conda + pip)
- Data/code sync between laptop and cluster
- PyCharm or Jupyter remote workflow

2. Enforce cluster-safe execution rules.
- Run heavy compute only on Slurm-allocated compute nodes.
- Keep login node for editing, `scp/rsync`, and job submission/monitoring.
- Keep `/home` for code, configs, notebooks, manifests, and standard lightweight experiment results.
- Prefer `$SCRATCH` for large datasets, caches, model weights, checkpoints, container images, raw predictions, and any large file or large directory.
- Keep code-generated logs, manifests, compact metrics, and small standard experiment outputs inside the project directory by default unless the user explicitly requests another location.
- If outputs are large or append-heavy, write them to scratch and keep only summaries, final tables, or small derived results in the project directory.

3. Choose execution mode.
- For quick debug: use `salloc` or `srun --pty bash`.
- For training/long runs: use `sbatch` and tail logs.
- For long-running training, require full-state checkpoints and latest-complete-checkpoint resume behavior.

4. Emit actionable commands, not generic advice.
- Provide exact command blocks with placeholder variables.
- Include one minimal example command and one production command.

5. Validate common failure points.
- Wrong partition/account/reservation
- Missing `module load`/`conda activate`
- Running training on login node
- GPU count mismatch between Slurm allocation and `torchrun`
- Training script saves model weights only and cannot exact-resume after timeout/preemption
- Home quota or scratch path issues
- In Codex/agent sessions, `$USER` may be `coder`; do not build `/scratch/$USER/...` or `/home/$USER/...` paths for submitted jobs unless that value is verified. Prefer explicit absolute user paths.
- Logs or run artifacts accidentally written outside the repository when the user expects project-local reproducibility records
- Large run artifacts accidentally written under `/home` when they should be on scratch

## Required References

- Use [palmetto-core.md](references/palmetto-core.md) for Slurm/login/sync patterns.
- Use [palmetto-ide-jupyter.md](references/palmetto-ide-jupyter.md) for PyCharm/Jupyter setup and SSH tunneling.

Load only the reference file needed for the user request.