# Tufts Hpc > Operate the Tufts University HPC cluster (SLURM) over SSH on the user's behalf: deploy code, submit / monitor / cancel jobs, run array-job experiment matrices, fetch results back, check quotas / partitions / GPU availability, and use the cluster as part of an iterate-loop. Use this skill whenever the user mentions running anything on the HPC / cluster / 集群 / 服务器 ("把它放到 hpc 上跑", "run this on the cluster", "submit to slurm", "在集群上训练"), asks about job status ("hpc 上跑得怎么样", "check my jobs", "任务跑完了吗"), storage quota, transferring files to/from the cluster, installing packages on the cluster, or anything involving sbatch / squeue / srun / sinfo / Tufts HPC — even casually. - Skill: `zhangyanbo/tufts-hpc` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add zhangyanbo/tufts-hpc` - Raw SKILL.md: https://api.skillmd.com/api/skills/zhangyanbo/tufts-hpc/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: Zhangyanbo (https://skillmd.com/u/zhangyanbo) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/zhangyanbo/tufts-hpc --- # Tufts HPC Operations Deploy, submit, monitor, and retrieve results on the Tufts HPC (SLURM) cluster over SSH, on the user's behalf. **Top priority: never do anything that could violate cluster policy or draw administrator attention** (see "Compliance red lines"). When in doubt, take the slower, by-the-book path — the user's account is not worth any shortcut. ## 0. Preflight — run this before any HPC work All connection details live on the local machine in `~/.config/tufts-hpc/config` (shell syntax, `KEY=value`). **This skill contains no account information**; the config file is the single source of truth. Start every HPC session with: ```bash bash /scripts/preflight.sh ``` (ssh may print warnings such as post-quantum key-exchange notices, possibly twice when a jump host is in the path — that is normal noise, not an error.) Branch on its output: - `READY host=... node=...` — connection works; proceed. `source` the config file to get: - `HPC_HOST` — ssh target (an alias or `user@host`). Use it everywhere: `ssh "$HPC_HOST" ...`, `rsync ... "$HPC_HOST":...` - `HPC_USER` — the user's Tufts UTLN - `HPC_DEFAULT_REMOTE_ROOT` — default deployment root *on the cluster* (a project's own remote path, if recorded in that project's CLAUDE.md, takes precedence). It is meant to stay a literal `~/...` or `/cluster/...` path. If preflight printed a `WARN:` about it (typically a local-machine path such as `/Users//...` or `/home//...`, caused by an unquoted `~` in the config expanding on the local side), **do not deploy to that value**: fix the config (quote it: `HPC_DEFAULT_REMOTE_ROOT="~/research"`) or use an explicit cluster path. - `HPC_RESEARCH_DIR` — research-storage path (may be empty = user has no lab storage yet; warn before placing large data anywhere) - `NO_CONFIG` — first use on this machine; run "Guided setup" below. - `NO_PASSWORDLESS` — key-based login is broken. **Stop all automated HPC operations.** Tell the user: passwordless SSH is a prerequisite for automation — set up a key (`ssh-keygen` + `ssh-copy-id `; if a jump host is involved, it needs a key too), then retry. Never type passwords for the user or try to work around authentication. ### Guided setup (only on NO_CONFIG) 1. Scan for candidates: `grep -B1 -A4 -i 'pax.tufts.edu' ~/.ssh/config`. Prefer an alias whose HostName is `login-prod.pax.tufts.edu` (the new cluster). `login.pax.tufts.edu` (no `-prod`) is the old cluster being retired — do not use it. 2. If nothing is found or multiple candidates exist, ask the user which Host to use and what their UTLN is. **Never guess account information.** 3. Test passwordless login: `ssh -o BatchMode=yes -o ConnectTimeout=10 'echo OK && hostname'`. 4. On success, write `~/.config/tufts-hpc/config` (template in the header of `scripts/preflight.sh`) and `chmod 600` it. Quote any `~` in path values (`HPC_DEFAULT_REMOTE_ROOT="~/research"`) — the file is `source`d locally and an unquoted `~` would silently become the local home. Also probe for research storage: `ssh 'ls -d /cluster/tufts/*/$USER 2>/dev/null'` and record it in `HPC_RESEARCH_DIR` if present. 5. On failure, follow NO_PASSWORDLESS above; do not write `HPC_PASSWORDLESS=yes`. ## 1. Compliance red lines (each one is a hard constraint) 1. **Login nodes are for light operations only**: ls / cat / squeue / sbatch / editing small files / small scp. Anything that burns CPU, memory, or heavy IO — installing packages, building conda/uv environments, extracting large archives, bulk-deleting big directories, compressing, data processing, running any program — **must go through a compute node**: ```bash # Interactive (for a sequence of manual steps; QOS caps at 4 hours) ssh -t "$HPC_HOST" 'srun -p batch -t 0-1:00:00 -c 4 --mem=8G --pty bash' # One-shot (for automation: wrap a single heavy command in srun) ssh "$HPC_HOST" 'srun -p batch -t 0-0:30:00 -c 2 --mem=4G bash -c "cd ~/proj && tar xzf data.tar.gz"' ``` All nodes share the same storage (home and /cluster), so work done on any compute node is visible everywhere — nodes differ only in compute power. "Borrowing a compute node for chores" has zero downside; don't hesitate. 2. **Every HPC action must trace back to an explicit user request.** Once the user says "run X on the HPC", the deploy → submit → monitor → retrieve chain for that task can run autonomously; actions outside that scope (touching other directories, cancelling unrelated jobs) are off-limits. 3. **Destructive operations require a confirmed list first**: bulk `rm`, overwriting existing remote results, `scancel` on jobs not submitted in this task. 4. **Rate-limit polling**: status-check loops at ≥ 60-second intervals (don't hammer squeue). Batch several remote commands into one ssh call (`ssh host 'cmd1; cmd2; cmd3'`) — fewer connections, and much lower latency through a jump host. 5. **Stay well below quota ceilings.** Before a large submission, check current load with `squeue --me | wc -l`. Per-user limits: batch+gpu combined ≤ 250 CPUs / 10 GPUs; preempt ≤ 1000 CPUs / 20 GPUs. Throttle arrays with `%N` (e.g. `%200`). 6. **No restricted data on the cluster** (HIPAA, FERPA, etc.). Never store or enter passwords on the user's behalf. ## 2. Status checks (when the user asks "how are my jobs doing?") Grab everything in one ssh call: ```bash ssh "$HPC_HOST" 'squeue --me; echo ---; sacct -X --starttime today -o JobID,JobName%20,State,Elapsed,ExitCode | tail -30' ``` | To see | Command (inside ssh) | |---|---| | Running / queued jobs | `squeue --me` | | Recent job outcomes (incl. failures) | `sacct -X --starttime -o JobID,JobName%20,State,Elapsed,ExitCode` | | Resource efficiency of a finished job | `seff ` | | Storage quota | `quota -s` (home hard limit is "30GB", shown as ~28611M — MiB units; full = writes blocked) | | Partition / node states | `sinfo` | | GPU availability | `module load hpctools && hpctools` (interactive menu; for automation use `sinfo -p gpu -o "%n %G %t"`) | | Job logs | `tail -50 /slurm-.out` (or the path set via `--output`) | Failure triage order: tail of the `.err` file → `sacct` State/ExitCode (`OOM` → more memory, `TIMEOUT` → more time, `NODE_FAIL`/`PREEMPTED` → just resubmit) → `seff` to see whether resources were undersized. ## 3. Task routing - **Deploying code / transferring files / setting up environments (conda, uv) / storage & quota issues** → read [references/deploy.md](references/deploy.md) first. - **Writing SLURM scripts / submitting jobs / array-job experiment matrices / choosing partitions / debugging jobs** → read [references/slurm.md](references/slurm.md) first. - Both at once (the common "run X on the HPC") → read both, execute in deploy → slurm order. - **Something in this skill turned out wrong or missing, or the user says "file that as an issue"** → §7 (draft an issue for the skill's repository; file only after the user confirms). ## 4. Using the HPC inside a loop (submit → wait → retrieve → iterate) Standard loop skeleton: 1. Deploy changed files (full dependency-chain check, see deploy.md), then submit (see slurm.md). 2. Wait in the background, polling at ≥ 60s intervals (scale up to 5–10 minutes for long jobs): ```bash ssh "$HPC_HOST" 'squeue --me -h | wc -l' # 0 = everything finished ``` When the queue drains, immediately classify with `sacct` — COMPLETED vs FAILED vs PREEMPTED. An empty queue does not mean success. 3. Retrieve results (create the local destination first — rsync does not create a missing parent directory, and then exits 0 having copied nothing): ```bash mkdir -p ./results && rsync -azP "$HPC_HOST":/results/ ./results/ && ls ./results | wc -l ``` Incremental and safe to re-run; the trailing count is the check that files actually arrived — never trust a clean exit alone (see deploy.md, "Transfer commands"). 4. Analyze locally → adjust code/parameters → back to 1. Re-run only the missing tasks using the idempotent submit pattern (slurm.md, "Idempotent submit pattern") — that pattern is what makes the whole loop safely re-entrant. ## 5. Key cluster facts (quick recall; details in references/) - New cluster login: `login-prod.pax.tufts.edu` (load-balanced across login-p01/02/03). Hostnames without `-prod` belong to the old, retiring cluster. - All public partitions cap at **2 days** (`2-00:00:00`). Partitions: `batch` (CPU only), `gpu` (must request `--gres`), `preempt` (most resources, but jobs can be preempted and are killed within ~30s — tasks must be idempotent / resumable). - Home `/cluster/home/$USER` has a 30GB quota; conda environments and large data belong in research storage, not home. - Conda: `module load miniforge/25.3.0` (there is **no** `module load python`). - Interactive session QOS: max 4 hours, max 1 GPU. ## 6. Off-campus access (knowledge, not an operation) This section is background to *explain* when the user asks about connection problems — it is not something this skill configures or performs on its own. The skill always just uses `$HPC_HOST`; whether that route goes through a VPN or a jump host is the user's local SSH client configuration, entirely outside the skill's operational scope. The login nodes are only reachable from the campus network. From off campus, the official route is the Tufts VPN. A common pain point: the VPN can be slow or flaky, and without it the cluster is unreachable. If the user controls an on-campus machine that is reachable from the internet (e.g. a lab server exposed via ngrok or similar), they can route SSH through it with `ProxyJump` in their local `~/.ssh/config`: ```ssh-config # Jump host: an on-campus machine the user controls Host my_jump_server HostName User Port # HPC via the jump host Host hpc HostName login-prod.pax.tufts.edu User ProxyJump my_jump_server ``` After `ssh-copy-id` to **both** the jump host and the HPC, `ssh hpc` is transparent, passwordless, and VPN-free — and everything in this skill works unchanged, since the jump is invisible above the SSH layer. If the user asks for this setup, help them edit their local ssh config; never store the jump host's details anywhere in a repository. ## 7. Reporting problems with this skill This skill lives in the public repository (directory `skills/tufts-hpc/`) and is maintained from real usage: when something in it turns out to be wrong or missing, that is worth an issue at . Typical triggers, noticed while doing a task: - a documented command failed or behaved differently from what the skill says (wrong module name, flag, path, limit); - a cluster fact here is stale (partition limits, quotas, hostnames, tool versions); - a gap that caused an avoidable detour — something you had to discover the hard way that the skill should have said up front. Procedure: 1. Finish the user's task first; collect evidence as you go (the exact command, its actual output / exit code, the skill file and section that was wrong or silent). 2. Draft the issue in English: title `tufts-hpc: `; body with which file/section, what the skill says, what actually happened, cluster-side evidence, and a suggested fix. Concise and concrete. 3. **De-sensitize the text — the repository is public.** No usernames / UTLN, ssh aliases, jump-host details, personal or lab paths: write ``, `$HPC_HOST`, `/cluster/home//...` instead. Job IDs and version numbers are fine. 4. **Show the draft to the user and file it only with their confirmation** (opening an issue is a public action): ```bash gh issue create --repo Zhangyanbo/hpc-skills --title "" --body-file <draft.md> ``` If `gh` is unavailable or the user prefers, hand them the draft to post themselves. Fixes are also welcome as pull requests (see the repository README).