# Palmetto Slurm Safe Guard

> Enforce safe Slurm job templates on Palmetto: keep /home for code and lightweight standard results, keep caches/checkpoints/models/data and other large files on /scratch, plus preflight safety checks, periodic low-noise GPU/process monitoring, resumable/atomic result saving, and signal-safe shutdown handling. Use whenever writing or updating sbatch scripts or long-running translation/training/inference jobs.

- Skill: `kwongfuk/palmetto-slurm-safe-guard` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add kwongfuk/palmetto-slurm-safe-guard`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/palmetto-slurm-safe-guard/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/palmetto-slurm-safe-guard

---


# Palmetto Slurm Safe Guard

## Use This Skill

Use this every time you write or edit a Slurm job script.
This is mandatory for long-running jobs (translation/training/inference/eval).
When the job is a training job, also apply `exact-training-resume-guard`.
Use `palmetto-slurm-workflow` alongside this skill for allocation/submission workflow details.

## Required Guards (Must Include)

- Safety validation before heavy work:
  - reject login node execution
  - check GPU tools (`nvidia-smi`)
  - check required GPU count
  - validate input file existence
- `/scratch` storage policy:
  - env/cache/model/data/container paths must be under `/scratch/$USER`
  - if the job is launched from a Codex/agent session, verify that `$USER` is the real Palmetto account before using it in scratch/home paths; if not, set an explicit `SCRATCH_ROOT=/scratch/<real_user>` and use that consistently
  - training checkpoints and any checkpoint-containing `output_dir` must live under `/scratch/$USER` (or `/scratch/<real_user>` in agent contexts), not `/home`
  - large outputs, large logs, raw predictions, intermediate artifacts, and resumable run directories must live under `/scratch/$USER`
  - keep lightweight code-generated logs, manifests, compact metrics, and standard small experiment results inside the project directory unless the user explicitly requests another location
  - if a framework mixes checkpoints with TensorBoard/event files inside one `output_dir`, put that `output_dir` on scratch and keep separate run logs under project-local `logs/`
- Model cache and offline policy:
  - pre-cache every model/tokenizer/processor repo used by the job under `/scratch/$USER/.hf_cache` before starting the main workload
  - run production training/eval/inference jobs in offline mode by default: `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, `HF_DATASETS_OFFLINE=1`
  - if offline mode fails due to missing files, fail fast and report `cache preload incomplete` instead of downloading during the run
- Checkpointed result saving:
  - periodic saves by rows/batches/time
  - atomic file write (`.tmp` then replace)
  - persist progress state JSON
- Exact training resume for training jobs:
  - save full training state, not model-only checkpoints
  - preserve optimizer / scheduler / RNG / framework state needed for exact resume
  - auto-detect the latest complete checkpoint and resume from it
  - skip incomplete checkpoints instead of resuming from partial files
- Signal-safe termination:
  - handle `SIGTERM` / `SIGINT`
  - force final save before exit
- Runtime monitoring without log spam:
  - background monitor loop every `MONITOR_INTERVAL` seconds (default 120)
  - emit concise GPU util/memory and key process lines
  - kill monitor in `trap ... EXIT`

## Slurm Template Checklist

1. Preflight
- `set -euo pipefail`
- node class check (login vs compute)
- `nvidia-smi` exists and GPU count meets requirement
- input/output paths validated

2. Scratch Paths
- `HF_HOME=/scratch/$USER/.hf_cache`
- `PIP_CACHE_DIR=/scratch/$USER/.pip_cache`
- env under `/scratch/$USER/envs/...`
- optional containers under `/scratch/$USER/containers`
- datasets, model weights, checkpoints, and large result directories under `/scratch/$USER/...`
- for Codex/agent-submitted jobs, prefer `SCRATCH_ROOT=/scratch/<real_user>` over raw `$USER` expansion if `$USER` may resolve to `coder`
- checkpoint trees and checkpoint-writing `output_dir` under `/scratch/$USER/...`
- repository keeps code, configs, manifests, lightweight logs, lightweight metrics, and standard small experiment results
- if artifacts are large, append-heavy, or likely to exceed home quota, place them on scratch and keep only small summaries or pointers in the repository
- pre-warm model repos into `HF_HOME` before the main Python command
- set offline defaults in job scripts: `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, `HF_DATASETS_OFFLINE=1`

3. Logging and Monitoring
- print one startup block: job id, model, input, output, limits
- monitor loop every 120s by default
- do not print per-step noisy diagnostics unless requested
- write `#SBATCH -o/-e` and code-generated run logs to project-local directories by default

4. Saving and Recovery
- pass save controls to Python entry:
  - `--save-every`
  - `--save-every-batches`
  - `--save-every-seconds`
  - `--progress-json`
- Python entry must save atomically and flush on signals
- For training jobs, checkpoint logic must satisfy `exact-training-resume-guard`; model-only saves do not count as resumable training checkpoints.

## Python Entry Requirements

- Add autosave policy driven by rows/batches/time.
- Save output CSV atomically.
- Save `<output>.progress.json` with current language/column/counters.
- Install signal handlers for graceful early termination with forced save.
- Process in batches and write translated content incrementally to memory + periodic disk flush.

## Default Parameters

- `MONITOR_INTERVAL=120`
- `SAVE_EVERY=200`
- `SAVE_EVERY_BATCHES=10`
- `SAVE_EVERY_SECONDS=120`
- `LOG_EVERY_BATCHES=10`

## Definition of Done

A job script is only done when all are true:

- preflight checks implemented
- scratch-only heavy paths enforced
- monitor loop exists and is cleanup-safe
- Python entry supports autosave and signal-safe final save
- training jobs support exact resume with full optimizer/scheduler/RNG/framework state
- output CSV and progress JSON are written during run (not only at end)
- code-generated logs and lightweight experiment artifacts stay inside the repository unless the user explicitly requested another path
- checkpoint trees and checkpoint-writing output directories stay under scratch
- large artifacts and large result directories stay under scratch
- model/tokenizer/processor repos are pre-cached under scratch and the run executes offline by default