Palmetto Conda Scratch Guard
Rules
- Treat
/home/$USER as the source tree area: code, configs, small manifests, and standard lightweight experiment results may live there.
- Treat
/scratch/$USER as the heavy-storage area: envs, caches, datasets, model weights, container images, checkpoints, raw intermediate outputs, and any large file or large directory go there.
- Create new environments under
/scratch/$USER/envs/<env_name>.
- Store Hugging Face model cache under
/scratch/$USER/.hf_cache.
- Store dataset downloads under
/scratch/$USER/data.
- Store model weights and large generated artifacts under
/scratch/$USER/models or another scratch-backed project path.
- Store container images under
/scratch/$USER/containers.
- Store pip cache under
/scratch/$USER/.pip_cache.
- In Codex/agent-run Slurm jobs, do not assume
$USER matches the real Palmetto account. If the session user can be coder, use an explicit absolute scratch root such as /scratch/<real_user> or a validated SCRATCH_ROOT variable instead of deriving critical paths from $USER.
- Do not create heavy Conda environments under
/home/$USER unless the user explicitly requests it.
- Do not place large caches, downloaded models, dataset shards, checkpoints, or other large artifacts under
/home/$USER.
- If an env path is missing, create
/scratch/$USER/envs first.
- Keep login node usage light; prefer compute allocation for heavy install/build workloads.
- Set
PYTHONNOUSERSITE=1 for env checks to avoid accidental package shadowing from ~/.local.
Workflow
- Run preflight checks:
bash scripts/palmetto_preflight.sh
- Create environment in scratch:
mkdir -p /scratch/$USER/envs
conda create -p /scratch/$USER/envs/<env_name> python=3.10 -y
- Activate by absolute path:
conda activate /scratch/$USER/envs/<env_name>
Install packages.
Set cache paths to scratch before model/data/container downloads:
mkdir -p /scratch/$USER/{.hf_cache,.pip_cache,data,containers,models}
export HF_HOME=/scratch/$USER/.hf_cache
export TRANSFORMERS_CACHE=$HF_HOME/transformers
export HUGGINGFACE_HUB_CACHE=$HF_HOME/hub
export PIP_CACHE_DIR=/scratch/$USER/.pip_cache
export PYTHONNOUSERSITE=1
Decision Guide
- If preflight reports
NODE_CLASS=login and task is heavy (large pip/conda builds, CUDA extensions, model runtime setup), move to an allocated compute node first (salloc or srun --pty bash).
- If task includes pulling large models/datasets/containers, run on compute allocation and keep all targets under
/scratch/$USER.
- If an experiment produces only standard summary artifacts such as final tables, plots, or compact metrics, they may stay with the code under
/home/$USER.
- If an output is large, resumable, or expensive to regenerate, keep it under
/scratch/$USER even if the final summarized result is later copied back to the project tree.
- If
nvidia-smi is missing or no GPUs are visible, check node class first. Login nodes often have no GPUs.
- If
torch.cuda.is_available() is false but GPUs exist, verify CUDA toolkit/driver compatibility and env package build.
References
- See
references/checklist.md for a compact troubleshooting checklist.
1---2name: palmetto-conda-scratch-guard3description: "Enforce Palmetto storage policy: keep /home for code, configs, and standard small experiment results, while placing Conda envs, caches, datasets, models, containers, checkpoints, and other large files under /scratch; run preflight checks for login-vs-compute node, GPU visibility, and CUDA tools/runtime before installs or training. Use when creating/updating envs, pulling HF models/datasets, preparing Apptainer images, debugging missing GPU/CUDA, or validating cluster node context."4metadata:5 source: https://skillmd.com/skills/kwongfuk/palmetto-conda-scratch-guard6---78# Palmetto Conda Scratch Guard910## Rules1112- Treat `/home/$USER` as the source tree area: code, configs, small manifests, and standard lightweight experiment results may live there.13- Treat `/scratch/$USER` as the heavy-storage area: envs, caches, datasets, model weights, container images, checkpoints, raw intermediate outputs, and any large file or large directory go there.14- Create new environments under `/scratch/$USER/envs/<env_name>`.15- Store Hugging Face model cache under `/scratch/$USER/.hf_cache`.16- Store dataset downloads under `/scratch/$USER/data`.17- Store model weights and large generated artifacts under `/scratch/$USER/models` or another scratch-backed project path.18- Store container images under `/scratch/$USER/containers`.19- Store pip cache under `/scratch/$USER/.pip_cache`.20- In Codex/agent-run Slurm jobs, do not assume `$USER` matches the real Palmetto account. If the session user can be `coder`, use an explicit absolute scratch root such as `/scratch/<real_user>` or a validated `SCRATCH_ROOT` variable instead of deriving critical paths from `$USER`.21- Do not create heavy Conda environments under `/home/$USER` unless the user explicitly requests it.22- Do not place large caches, downloaded models, dataset shards, checkpoints, or other large artifacts under `/home/$USER`.23- If an env path is missing, create `/scratch/$USER/envs` first.24- Keep login node usage light; prefer compute allocation for heavy install/build workloads.25- Set `PYTHONNOUSERSITE=1` for env checks to avoid accidental package shadowing from `~/.local`.2627## Workflow28291. Run preflight checks:3031```bash32bash scripts/palmetto_preflight.sh33```34352. Create environment in scratch:3637```bash38mkdir -p /scratch/$USER/envs39conda create -p /scratch/$USER/envs/<env_name> python=3.10 -y40```41423. Activate by absolute path:4344```bash45conda activate /scratch/$USER/envs/<env_name>46```47484. Install packages.49505. Set cache paths to scratch before model/data/container downloads:5152```bash53mkdir -p /scratch/$USER/{.hf_cache,.pip_cache,data,containers,models}54export HF_HOME=/scratch/$USER/.hf_cache55export TRANSFORMERS_CACHE=$HF_HOME/transformers56export HUGGINGFACE_HUB_CACHE=$HF_HOME/hub57export PIP_CACHE_DIR=/scratch/$USER/.pip_cache58export PYTHONNOUSERSITE=159```6061## Decision Guide6263- If preflight reports `NODE_CLASS=login` and task is heavy (large pip/conda builds, CUDA extensions, model runtime setup), move to an allocated compute node first (`salloc` or `srun --pty bash`).64- If task includes pulling large models/datasets/containers, run on compute allocation and keep all targets under `/scratch/$USER`.65- If an experiment produces only standard summary artifacts such as final tables, plots, or compact metrics, they may stay with the code under `/home/$USER`.66- If an output is large, resumable, or expensive to regenerate, keep it under `/scratch/$USER` even if the final summarized result is later copied back to the project tree.67- If `nvidia-smi` is missing or no GPUs are visible, check node class first. Login nodes often have no GPUs.68- If `torch.cuda.is_available()` is false but GPUs exist, verify CUDA toolkit/driver compatibility and env package build.6970## References7172- See `references/checklist.md` for a compact troubleshooting checklist.