# Northwestern Quest > Operate the Northwestern University Quest HPC cluster (SLURM) over SSH on the user's behalf: deploy code, submit / monitor / cancel jobs, manage GPU allocations, fetch results back, check quotas / partitions / GPU availability, and use the cluster as part of an iterate-loop. Use this skill whenever the user mentions running anything on Quest / the HPC / cluster ("run this on Quest", "submit to slurm", "check my quest jobs", "在 quest 上跑"), asks about job status, storage quota, transferring files to/from the cluster, installing packages on the cluster, or anything involving sbatch / squeue / srun / sinfo / Quest — even casually. - Skill: `zhangyanbo/northwestern-quest` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add zhangyanbo/northwestern-quest` - Raw SKILL.md: https://api.skillmd.com/api/skills/zhangyanbo/northwestern-quest/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: Zhangyanbo (https://skillmd.com/u/zhangyanbo) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/zhangyanbo/northwestern-quest --- # Northwestern Quest HPC Operations Deploy, submit, monitor, and retrieve results on Quest (SLURM) over SSH, on the user's behalf. **Top priority: never do anything that could violate cluster policy or draw administrator attention** (see "Compliance red lines"). When in doubt, take the slower, by-the-book path — the user's account is not worth any shortcut. ## 0. Preflight — run this before any Quest work All connection details live on the local machine in `~/.config/quest-hpc/config` (shell syntax, `KEY=value`). **This skill contains no account information**; the config file is the single source of truth. Start every Quest session with: ```bash bash /scripts/preflight.sh ``` Branch on its output: - `READY host=... node=...` — connection works; proceed. `source` the config file to get: - `QUEST_HOST` — ssh target (an alias or `user@host`). Use it everywhere: `ssh "$QUEST_HOST" ...`, `rsync ... "$QUEST_HOST":...` - `QUEST_NETID` — the user's Northwestern NetID - `QUEST_ALLOCATION_ROOT` — default project storage root, e.g. `/projects/` or a subdirectory of it the user works in (a project's own remote path, if recorded in that project's CLAUDE.md, takes precedence) - `QUEST_SCRATCH_ROOT` — `/scratch/` (auto-purged; see storage notes) - `NO_CONFIG` — first use on this machine; run "Guided setup" below. - `NO_PASSWORDLESS` — key-based login is broken. **Stop all automated Quest operations.** Tell the user: passwordless SSH is a prerequisite for automation — set up a key (`ssh-keygen` + `ssh-copy-id `), then retry. Never type passwords for the user or try to work around authentication. ### Guided setup (only on NO_CONFIG) 1. Scan for candidates: `grep -B1 -A4 -i 'quest.northwestern.edu' ~/.ssh/config`. 2. If nothing is found or multiple candidates exist, ask the user which Host to use and what their NetID is. **Never guess account information.** 3. Test passwordless login: `ssh -o BatchMode=yes -o ConnectTimeout=10 'echo OK && hostname'`. 4. On success, write `~/.config/quest-hpc/config` (template in the header of `scripts/preflight.sh`) and `chmod 600` it. Also probe for the project allocation: `ssh 'groups'` — allocation IDs show up as groups like `p12345`/`b1234`. Confirm the directory exists (`ls -d /projects/`) and record it in `QUEST_ALLOCATION_ROOT`. Note `/projects/` is **shared by all members of the allocation**; a per-user subdirectory under it is a lab convention, not guaranteed — ask the user where their space is. 5. On failure, follow NO_PASSWORDLESS above; do not write `QUEST_PASSWORDLESS=yes`. ## 1. Compliance red lines (each one is a hard constraint) 1. **Login nodes are for light operations only**: ls / cat / squeue / sbatch / editing small files / small scp. Anything that burns CPU, memory, or heavy IO — installing packages, building conda/mamba environments, extracting large archives, bulk-deleting big directories, data processing, running any program — **must go through a compute node**, either an interactive allocation (`salloc`) or a batch job (`sbatch`). 2. **Every Quest action must trace back to an explicit user request.** Once the user says "run X on Quest", the deploy → submit → monitor → retrieve chain for that task can run autonomously; actions outside that scope (touching other directories, cancelling unrelated jobs) are off-limits. 3. **Destructive operations require a confirmed list first**: bulk `rm`, overwriting existing remote results, `scancel` on jobs not submitted in this task, killing an interactive/placeholder allocation another task may still need. 4. **Rate-limit polling**: status-check loops at ≥ 60-second intervals (don't hammer `squeue`). Batch several remote commands into one ssh call (`ssh host 'cmd1; cmd2; cmd3'`) — fewer connections, lower latency. 5. **Stay well below quota ceilings.** Before a large submission, check current allocation load with `squeue -u | wc -l`. GPU allocations (`QOSMaxGRESPerUser` and similar) are limited per user — too many long-lived interactive/placeholder jobs blocks new `sbatch` submissions with that pending reason. 6. **No restricted data on the cluster** (HIPAA, FERPA, etc.). Never store or enter passwords/API keys (e.g. `WANDB_API_KEY`, `HF_TOKEN`) on the user's behalf — those belong in a project `.env`. ## 2. Status checks (when the user asks "how are my jobs doing?") Grab everything in one ssh call: ```bash ssh "$QUEST_HOST" 'squeue -u '"$QUEST_NETID"'; echo ---; sacct -X --starttime today -o JobID,JobName%20,State,Elapsed,ExitCode | tail -30' ``` | To see | Command (inside ssh) | |---|---| | Running / queued jobs | `squeue -u ` | | Recent job outcomes (incl. failures) | `sacct -X --starttime -o JobID,JobName%20,State,Elapsed,ExitCode` | | Resource efficiency of a finished job | `seff ` | | Storage quota | check the allocation's quota per Quest's storage documentation; quotas are set per PI allocation, not per user | | Partition / node states | `sinfo` | | GPU availability | `sinfo -p gengpu -o "%n %G %t"` (adjust partition name to the allocation's GPU partition) | | Job logs | `tail -50 /slurm-.out` (or the path set via `--output`) | | Live GPU utilization on an allocated node | `ssh "$QUEST_HOST" "ssh 'nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv,noheader'"` | Failure triage order: tail of the `.err` file → `sacct` State/ExitCode (`OOM` → more memory, `TIMEOUT` → more time, `NODE_FAIL`/`PREEMPTED` → just resubmit) → `seff` to see whether resources were undersized. A `RUNNING` job with 0% GPU utilization and near-zero memory is very likely an idle interactive placeholder, not active training — always verify GPU utilization before assuming a job is doing useful work. ## 3. Task routing - **Deploying code / transferring files / setting up environments (conda, uv) / storage & quota issues** → read [references/deploy.md](references/deploy.md) first. - **Writing SLURM scripts / submitting jobs / GPU-type-aware requests / debugging jobs / long-running detached processes** → read [references/slurm.md](references/slurm.md) first. - Both at once (the common "run X on Quest") → read both, execute in deploy → slurm order. - **Something in this skill turned out wrong or missing, or the user says "file that as an issue"** → §7 (draft an issue for the skill's repository; file only after the user confirms). ## 4. Using Quest inside a loop (submit → wait → retrieve → iterate) Standard loop skeleton: 1. Deploy changed files (full dependency-chain check, see deploy.md), then submit (see slurm.md). 2. Wait in the background, polling at ≥ 60s intervals (scale up to 5–10 minutes for long jobs): ```bash ssh "$QUEST_HOST" 'squeue -u '"$QUEST_NETID"' -h | wc -l' # 0 = everything finished ``` When the queue drains, immediately classify with `sacct` — COMPLETED vs FAILED vs PREEMPTED/TIMEOUT. An empty queue does not mean success. 3. Retrieve results (create the local destination first — rsync does not create a missing parent directory, and then exits 0 having copied nothing): ```bash mkdir -p ./results && rsync -azP "$QUEST_HOST":/results/ ./results/ && ls ./results | wc -l ``` Incremental and safe to re-run; the trailing count is the check that files actually arrived — never trust a clean exit alone (see deploy.md, "Transfer commands"). 4. Analyze locally → adjust code/parameters → back to 1. Re-run only the missing tasks using the idempotent submit pattern (slurm.md, "Idempotent submit pattern") — that pattern is what makes the whole loop safely re-entrant. ## 5. Key cluster facts (quick recall; details in references/) - Scheduler: SLURM, reached over SSH from a login node. - General-access partitions are `short`/`normal`/`long` (walltime tiers) plus `gengpu` for GPUs (mixed generations, e.g. A100 and H100, selected via `--gres=gpu:a100:1` etc.); buy-in allocations have their own `b####` partitions. Verify live with `sinfo -o "%P %l %D %c %m %G"`. - Durable project storage lives under `/projects//` — a directory **shared by all members of the allocation**, sized per the PI's storage allocation. - `/scratch//` exists for temporary large outputs but is **auto-purged** (commonly after ~30 days) — never treat it as durable. - Conda/mamba environments must be explicitly activated in non-interactive shells (SSH / SLURM scripts) — do not rely on `.bashrc` being sourced; source the cluster's mamba/conda init script directly. - `tmux` may not be available on Quest compute nodes (check with `command -v tmux`); if absent, use `nohup ... log 2>&1 & disown` for long-running detached processes instead. ## 6. Off-campus access (knowledge, not an operation) This section is background to *explain* when the user asks about connection problems — it is not something this skill configures or performs on its own. The skill always just uses `$QUEST_HOST`; whether that route needs a VPN is the user's local SSH client configuration, entirely outside the skill's operational scope. If off-campus access requires Northwestern's VPN, the same "install a key, set `HostName` to the login endpoint" recipe applies — check current Northwestern IT documentation for the up-to-date VPN/SSH requirements, since these evolve independently of this skill. ## 7. Reporting problems with this skill This skill lives in the public repository (directory `skills/northwestern-quest/`) and is maintained from real usage: when something in it turns out to be wrong or missing, that is worth an issue at . Typical triggers, noticed while doing a task: - a documented command failed or behaved differently from what the skill says (wrong module name, flag, path, limit); - a cluster fact here is stale (partition limits, quotas, hostnames, tool versions); - a gap that caused an avoidable detour — something you had to discover the hard way that the skill should have said up front. Procedure: 1. Finish the user's task first; collect evidence as you go (the exact command, its actual output / exit code, the skill file and section that was wrong or silent). 2. Draft the issue in English: title `northwestern-quest: `; body with which file/section, what the skill says, what actually happened, cluster-side evidence, and a suggested fix. Concise and concrete. 3. **De-sensitize the text — the repository is public.** No usernames / NetID, ssh aliases, jump-host details, personal or lab paths: write ``, `$QUEST_HOST`, `/projects//...` instead. Job IDs and version numbers are fine. 4. **Show the draft to the user and file it only with their confirmation** (opening an issue is a public action): ```bash gh issue create --repo Zhangyanbo/hpc-skills --title "" --body-file <draft.md> ``` If `gh` is unavailable or the user prefers, hand them the draft to post themselves. Fixes are also welcome as pull requests (see the repository README).