Palmetto Slurm Safe Guard

Palmetto Slurm Safe Guard

Use This Skill

Use this every time you write or edit a Slurm job script. This is mandatory for long-running jobs (translation/training/inference/eval). When the job is a training job, also apply exact-training-resume-guard. Use palmetto-slurm-workflow alongside this skill for allocation/submission workflow details.

Required Guards (Must Include)

• Safety validation before heavy work:
  • reject login node execution
  • check GPU tools (nvidia-smi)
  • check required GPU count
  • validate input file existence
• /scratch storage policy:
  • env/cache/model/data/container paths must be under /scratch/$USER
  • if the job is launched from a Codex/agent session, verify that $USER is the real Palmetto account before using it in scratch/home paths; if not, set an explicit SCRATCH_ROOT=/scratch/<real_user> and use that consistently
  • training checkpoints and any checkpoint-containing output_dir must live under /scratch/$USER (or /scratch/<real_user> in agent contexts), not /home
  • large outputs, large logs, raw predictions, intermediate artifacts, and resumable run directories must live under /scratch/$USER
  • keep lightweight code-generated logs, manifests, compact metrics, and standard small experiment results inside the project directory unless the user explicitly requests another location
  • if a framework mixes checkpoints with TensorBoard/event files inside one output_dir, put that output_dir on scratch and keep separate run logs under project-local logs/
• Model cache and offline policy:
  • pre-cache every model/tokenizer/processor repo used by the job under /scratch/$USER/.hf_cache before starting the main workload
  • run production training/eval/inference jobs in offline mode by default: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1
  • if offline mode fails due to missing files, fail fast and report cache preload incomplete instead of downloading during the run
• Checkpointed result saving:
  • periodic saves by rows/batches/time
  • atomic file write (.tmp then replace)
  • persist progress state JSON
• Exact training resume for training jobs:
  • save full training state, not model-only checkpoints
  • preserve optimizer / scheduler / RNG / framework state needed for exact resume
  • auto-detect the latest complete checkpoint and resume from it
  • skip incomplete checkpoints instead of resuming from partial files
• Signal-safe termination:
  • handle SIGTERM / SIGINT
  • force final save before exit
• Runtime monitoring without log spam:
  • background monitor loop every MONITOR_INTERVAL seconds (default 120)
  • emit concise GPU util/memory and key process lines
  • kill monitor in trap ... EXIT

Slurm Template Checklist

1. Preflight

• set -euo pipefail
• node class check (login vs compute)
• nvidia-smi exists and GPU count meets requirement
• input/output paths validated

2. Scratch Paths

• HF_HOME=/scratch/$USER/.hf_cache
• PIP_CACHE_DIR=/scratch/$USER/.pip_cache
• env under /scratch/$USER/envs/...
• optional containers under /scratch/$USER/containers
• datasets, model weights, checkpoints, and large result directories under /scratch/$USER/...
• for Codex/agent-submitted jobs, prefer SCRATCH_ROOT=/scratch/<real_user> over raw $USER expansion if $USER may resolve to coder
• checkpoint trees and checkpoint-writing output_dir under /scratch/$USER/...
• repository keeps code, configs, manifests, lightweight logs, lightweight metrics, and standard small experiment results
• if artifacts are large, append-heavy, or likely to exceed home quota, place them on scratch and keep only small summaries or pointers in the repository
• pre-warm model repos into HF_HOME before the main Python command
• set offline defaults in job scripts: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1

3. Logging and Monitoring

• print one startup block: job id, model, input, output, limits
• monitor loop every 120s by default
• do not print per-step noisy diagnostics unless requested
• write #SBATCH -o/-e and code-generated run logs to project-local directories by default

4. Saving and Recovery

• pass save controls to Python entry:
  • --save-every
  • --save-every-batches
  • --save-every-seconds
  • --progress-json
• Python entry must save atomically and flush on signals
• For training jobs, checkpoint logic must satisfy exact-training-resume-guard; model-only saves do not count as resumable training checkpoints.

Python Entry Requirements

• Add autosave policy driven by rows/batches/time.
• Save output CSV atomically.
• Save <output>.progress.json with current language/column/counters.
• Install signal handlers for graceful early termination with forced save.
• Process in batches and write translated content incrementally to memory + periodic disk flush.

Default Parameters

• MONITOR_INTERVAL=120
• SAVE_EVERY=200
• SAVE_EVERY_BATCHES=10
• SAVE_EVERY_SECONDS=120
• LOG_EVERY_BATCHES=10

Definition of Done

A job script is only done when all are true:

• preflight checks implemented
• scratch-only heavy paths enforced
• monitor loop exists and is cleanup-safe
• Python entry supports autosave and signal-safe final save
• training jobs support exact resume with full optimizer/scheduler/RNG/framework state
• output CSV and progress JSON are written during run (not only at end)
• code-generated logs and lightweight experiment artifacts stay inside the repository unless the user explicitly requested another path
• checkpoint trees and checkpoint-writing output directories stay under scratch
• large artifacts and large result directories stay under scratch
• model/tokenizer/processor repos are pre-cached under scratch and the run executes offline by default