Path Storage Policy

Path Storage Policy

Apply this skill whenever a task creates, downloads, writes, moves, or plans files and directories.

Core Policy

• Treat /home/$USER as the durable source tree area.
• Treat /scratch/$USER as the heavy-storage and high-churn area.
• Keep code, configs, notebooks, manifests, small logs, and standard lightweight experiment results under /home/$USER.
• Keep environments, caches, datasets, model weights, checkpoints, container images, raw intermediates, append-heavy outputs, and any large file or large directory under /scratch/$USER.
• If an artifact may grow substantially, is expensive to regenerate, or may exceed home quota, place it on scratch by default.
• If only a compact final summary is needed, keep the summary in the project tree and leave the heavy upstream artifacts on scratch.

Decision Rules

1. Put it in /home/$USER when it is primarily:

• source code
• configuration
• notebooks
• lightweight documentation
• small manifests or metadata
• standard small experiment outputs such as final tables, plots, and compact metrics

2. Put it in /scratch/$USER when it is primarily:

• a cache
• a virtual environment or Conda environment
• downloaded data
• downloaded or generated model weights
• checkpoints or resumable run state
• raw predictions, intermediate outputs, or staged preprocessing results
• a container image
• a large result directory
• a file or directory that is large, append-heavy, temporary-but-costly, or high churn

3. If uncertain, prefer this split:

• small human-reviewed deliverables in /home/$USER
• heavy machine-generated artifacts in /scratch/$USER

Planning Workflow

1. Identify every path the task will read or write.
2. Classify each path as one of:

• source/config
• cache/env
• dataset/model
• checkpoint/intermediate
• final result
• log/record

3. Place source/config, logs, records, and small standard outputs under /home/$USER.
4. Place cache/env, dataset/model, checkpoints, intermediates, and large outputs under /scratch/$USER.
5. When a final result has both heavy and lightweight forms, store:

• heavy primary artifact on scratch
• lightweight summary or pointer in the project tree

6. Use absolute paths when the runtime environment may rewrite $USER or $HOME.

Recommended Layout

/home/$USER/code/<project>/
/home/$USER/code/<project>/configs/
/home/$USER/code/<project>/notebooks/
/home/$USER/code/<project>/results/
/home/$USER/code/<project>/logs/

/scratch/$USER/envs/
/scratch/$USER/.cache/
/scratch/$USER/data/
/scratch/$USER/models/
/scratch/$USER/containers/
/scratch/$USER/runs/<project>/
/scratch/$USER/checkpoints/<project>/

Operational Rules

• Before downloads or installs, redirect caches to scratch.
• Before long runs, ensure checkpoint and large-output directories point to scratch.
• Do not silently write large artifacts under /home/$USER.
• If a tool defaults to ~/.cache, ~/.conda, or project-local heavy outputs, override it to scratch when practical.
• If the task is on shared compute or HPC, assume scratch is the default destination for heavy artifacts unless the user explicitly asks otherwise.

Examples

• A git repository, training script, and final markdown report belong under /home/$USER.
• A Conda env, Hugging Face cache, dataset shard, and model checkpoint belong under /scratch/$USER.
• A 200 MB prediction dump belongs under /scratch/$USER, while a 20 KB summary CSV can stay in the repository.
• A long-running experiment may keep compact metrics under the project results/ directory while storing checkpoints and raw batch outputs on scratch.