# Palmetto Slurm Fit Check

> Check Palmetto Slurm CPU/GPU/memory fit before submitting or editing GPU jobs. Use when a request involves sbatch scripts, GPU training jobs, H100/A100/H200 nodes, or deciding whether a current CPU request is too high for available node fragmentation.

- Skill: `kwongfuk/palmetto-slurm-fit-check` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add kwongfuk/palmetto-slurm-fit-check`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/palmetto-slurm-fit-check/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/palmetto-slurm-fit-check

---


# Palmetto Slurm Fit Check

Use this skill before submitting or retuning a Palmetto GPU job.

Goal:
- inspect the current node fragmentation
- decide whether the requested CPU/GPU/memory tuple can start now
- recommend a lower CPU request when GPUs are available but CPUs are fragmented
- tell the user whether to submit now, lower CPUs, or wait

## When To Use

Trigger this skill when the user asks to:
- submit or resubmit a Palmetto GPU job
- tune `#SBATCH --cpus-per-task` for H100/A100/H200 jobs
- check whether a job can run now
- find a better CPU/GPU ratio before `sbatch`
- explain why a GPU job is pending even though some GPUs look free

## Workflow

1. Read the target request from the `sbatch` script when possible.
2. Run the fit checker before any submission:

```bash
python /home/gguo/.codex/skills/palmetto-slurm-fit-check/scripts/check_slurm_fit.py \
  --sbatch-script /path/to/job.sbatch
```

3. Interpret the verdict:
- `CAN_RUN_NOW`: the current request fits at least one node now
- `LOWER_CPU_AND_RESUBMIT`: GPUs and memory fit, but CPUs are too fragmented; lower `--cpus-per-task`
- `WAIT_OR_REDUCE_MEMORY`: GPUs fit, memory does not
- `WAIT_FOR_GPUS`: not enough free GPUs of the requested type

4. Do not submit immediately when the verdict is `LOWER_CPU_AND_RESUBMIT`.
- Patch the script first.
- Prefer the script's recommended CPU count over the original request.

5. After patching, validate with:

```bash
sbatch --test-only /path/to/job.sbatch
```

6. Only then submit or resubmit the job.

## Heuristics

- Treat CPU fragmentation as a first-class blocker, not just GPU count.
- Default conservative CPU targets:
  - `h100`, `h200`, `a100`, `v100`: about `12 CPU / GPU`
  - `p100`, `k40`, `k20`, `gtx_1080`: about `8 CPU / GPU`
- Round CPU recommendations down in steps of 4 unless the user asks otherwise.
- If the script can fit with fewer CPUs on a partially free node, tell the user that explicitly instead of keeping an inflated CPU request.

## Notes

- The checker handles the local `scontrol` shared-library issue automatically by adding a temporary `libhistory.so.7` shim when needed.
- The checker is intended for single-node GPU jobs. If a script requests multiple nodes, say that the fit result is only approximate and inspect manually.
- Keep the decision user-visible: explain whether the job can run now and why.