# Palmetto Conda Scratch Guard

> Enforce Palmetto storage policy: keep /home for code, configs, and standard small experiment results, while placing Conda envs, caches, datasets, models, containers, checkpoints, and other large files under /scratch; run preflight checks for login-vs-compute node, GPU visibility, and CUDA tools/runtime before installs or training. Use when creating/updating envs, pulling HF models/datasets, preparing Apptainer images, debugging missing GPU/CUDA, or validating cluster node context.

- Skill: `kwongfuk/palmetto-conda-scratch-guard` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add kwongfuk/palmetto-conda-scratch-guard`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/palmetto-conda-scratch-guard/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/palmetto-conda-scratch-guard

---


# Palmetto Conda Scratch Guard

## Rules

- Treat `/home/$USER` as the source tree area: code, configs, small manifests, and standard lightweight experiment results may live there.
- Treat `/scratch/$USER` as the heavy-storage area: envs, caches, datasets, model weights, container images, checkpoints, raw intermediate outputs, and any large file or large directory go there.
- Create new environments under `/scratch/$USER/envs/<env_name>`.
- Store Hugging Face model cache under `/scratch/$USER/.hf_cache`.
- Store dataset downloads under `/scratch/$USER/data`.
- Store model weights and large generated artifacts under `/scratch/$USER/models` or another scratch-backed project path.
- Store container images under `/scratch/$USER/containers`.
- Store pip cache under `/scratch/$USER/.pip_cache`.
- In Codex/agent-run Slurm jobs, do not assume `$USER` matches the real Palmetto account. If the session user can be `coder`, use an explicit absolute scratch root such as `/scratch/<real_user>` or a validated `SCRATCH_ROOT` variable instead of deriving critical paths from `$USER`.
- Do not create heavy Conda environments under `/home/$USER` unless the user explicitly requests it.
- Do not place large caches, downloaded models, dataset shards, checkpoints, or other large artifacts under `/home/$USER`.
- If an env path is missing, create `/scratch/$USER/envs` first.
- Keep login node usage light; prefer compute allocation for heavy install/build workloads.
- Set `PYTHONNOUSERSITE=1` for env checks to avoid accidental package shadowing from `~/.local`.

## Workflow

1. Run preflight checks:

```bash
bash scripts/palmetto_preflight.sh
```

2. Create environment in scratch:

```bash
mkdir -p /scratch/$USER/envs
conda create -p /scratch/$USER/envs/<env_name> python=3.10 -y
```

3. Activate by absolute path:

```bash
conda activate /scratch/$USER/envs/<env_name>
```

4. Install packages.

5. Set cache paths to scratch before model/data/container downloads:

```bash
mkdir -p /scratch/$USER/{.hf_cache,.pip_cache,data,containers,models}
export HF_HOME=/scratch/$USER/.hf_cache
export TRANSFORMERS_CACHE=$HF_HOME/transformers
export HUGGINGFACE_HUB_CACHE=$HF_HOME/hub
export PIP_CACHE_DIR=/scratch/$USER/.pip_cache
export PYTHONNOUSERSITE=1
```

## Decision Guide

- If preflight reports `NODE_CLASS=login` and task is heavy (large pip/conda builds, CUDA extensions, model runtime setup), move to an allocated compute node first (`salloc` or `srun --pty bash`).
- If task includes pulling large models/datasets/containers, run on compute allocation and keep all targets under `/scratch/$USER`.
- If an experiment produces only standard summary artifacts such as final tables, plots, or compact metrics, they may stay with the code under `/home/$USER`.
- If an output is large, resumable, or expensive to regenerate, keep it under `/scratch/$USER` even if the final summarized result is later copied back to the project tree.
- If `nvidia-smi` is missing or no GPUs are visible, check node class first. Login nodes often have no GPUs.
- If `torch.cuda.is_available()` is false but GPUs exist, verify CUDA toolkit/driver compatibility and env package build.

## References

- See `references/checklist.md` for a compact troubleshooting checklist.