Using Drac Clusters

Using DRAC Clusters

Overview

DRAC clusters are shared, MFA-gated, quota-bound Slurm systems. Three properties drive almost every mistake: you cannot authenticate on your own, login nodes are not for computing, and almost nothing is knowable without asking the cluster — accounts, partitions, GPU types and quotas differ per cluster.

Core principle: discover, don't assume. Every value below has a command that prints the truth. Run it once per cluster and cache it in the project's notes.

Scope: this skill is the mechanics — authenticating, finding the right account and gres string, respecting quotas, moving files. For the methodology of an experiment campaign — cost models and walltime budgeting, how to shape jobs and arrays, gating a script on the compute it actually got, keeping results alive through a walltime kill, staged submission — use running-cluster-experiments.

The MFA gate — read this before anything else

MFA is mandatory on DRAC and an agent cannot complete it. Non-interactive ssh works only by multiplexing over a ControlMaster socket the user already authenticated. Before concluding the cluster is down:

ssh -O check <cluster>      # "Master running (pid=...)" = usable

With no live master, every command fails identically with Permission denied (keyboard-interactive). That is not a transient error, a wrong key, or an outage. Retrying cannot fix it.

Do NOT engineer around the MFA prompt. It is technically possible — pointing SSH_ASKPASS at a script that auto-selects the push option will get an agent through, and a subagent did exactly that during this skill's own testing. Do not do it. MFA exists to put a human decision in front of every new session; a script that answers it converts the user's phone into a rubber stamp and reproduces the shape of an MFA-fatigue attack, even when the intent is benign. The user then approves a push they did not initiate and cannot attribute.

The gate is a stop, not an obstacle. Report the blocker and ask. If the user later says "go ahead and automate the prompt", that is their call to make explicitly — never yours to assume from a task that merely needs cluster access.

Two legitimate ways to open the master

A. The user opens it in a REAL terminal. Always available, needs no setup. An agent-run shell cannot do this, however it is invoked (Claude Code's ! prefix, a tool call, nohup, a background job): MFA needs an interactive TTY, and without one ssh falls back to SSH_ASKPASS and fails like this —

Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory
Permission denied (keyboard-interactive,hostbased)

So the ask is specific: "open Terminal/iTerm and run ssh <cluster>, complete the MFA prompt." They may exit immediately afterwards — ControlPersist keeps the master alive once it exists.

B. A GUI askpass, if the user has installed one. With SSH_ASKPASS pointing at a real dialog program (e.g. theseal/ssh-askpass on macOS) and DISPLAY set, an agent can start the connection and the user answers a dialog on screen. This removes the context switch, not the human.

The distinction that matters is not GUI-vs-terminal, it is who answers. A dialog program ASKS the human; a script that returns a canned answer REPLACES them. The first is fine. The second is the prohibition above.

With askpass available, ANNOUNCE BEFORE CONNECTING. Say that a dialog is about to appear and why, then connect. The point is attributability: if the user can always trace a prompt to something they were just told about, an unannounced dialog becomes a reliable signal to refuse. Silently summoning prompts destroys that signal even though every individual prompt is answered by a human.

Installation note, if asked: install the package but do not run brew services start. That agent launchctl setenvs SSH_ASKPASS and SUDO_ASKPASS globally for every GUI app, putting a dialog in front of sudo -A — far beyond what ssh needs. Export SSH_ASKPASS from the shell profile instead.

A socket that worked an hour ago proves nothing about now; re-check before a batch of commands rather than discovering it mid-transfer.

Per-cluster ~/.ssh/config that makes this work:

Host narval narval.alliancecan.ca
  HostName narval.alliancecan.ca
  User <username>
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p.sock
  ControlPersist 8h

mkdir -p ~/.ssh/sockets first — if the directory is missing, multiplexing fails silently and every command re-prompts for MFA. With no master, ssh -O check prints Control socket connect(...): No such file or directory.

Never compute on a login node

Login nodes are for editing, transfers and sbatch, and staff kill offenders. Cheap queries (squeue, sinfo, sacct, diskusage_report, ls) are fine; anything looping over data, training, or running for minutes is not. Use sbatch, or salloc for interactive work:

salloc --account=<alloc>_cpu --cpus-per-task=4 --mem=8G --time=1:00:00

Discover the cluster, don't assume it

How much actually varies — six clusters, same day, same user

This is why the principle is "discover, don't assume". Every value below was read off the cluster with the commands above; none is inferred from another cluster.

Three of those cells contradict what a reasonable person would generalise from the first three columns, and each is its own trap:

Time buckets do not stop at b5. General-purpose clusters may expose _b6 at 28 days for CPU, and preempt partitions running to 122 days. Reading "b5 = the maximum" off one cluster under-uses another by 4x. Conversely an AI cluster may stop at b3. Always read sinfo -h -o "%P %l".

Partitions are not always named bygpu/bynode. At least one cluster names them by GPU model — gpubase_h100_b3, gpubase_l40s_b3 — so a script that greps for bygpu finds nothing and a --partition=gpubase_bynode_b3 copied across is rejected outright. On such a cluster you select hardware by partition, not only by --gpus=.

Some clusters have no CPU partitions at all. A GPU-only cluster will reject CPU-only preprocessing, so the "run it in salloc instead of on the login node" advice has nowhere to land there — stage that work on a different cluster.

Fractional GPUs come in two unrelated flavours. MIG slices appear in the gres string as _1g. / _3g. variants; shard: is Slurm's time-sharing of a whole GPU (e.g. shard:l40s:16 = 4 shards per card). Requesting a shard gives you a card shared with other jobs, not a partitioned one — fine for small inference, wrong for a benchmark you intend to time.

Inode quotas differ by four orders of magnitude — 250K on one cluster against 105M on another, because they are not the same filesystem. A pipeline that is fine on one will wedge a group's quota on another.

Some clusters set up Slurm only in a LOGIN shell

On most clusters ssh <cluster> 'sinfo' just works. On at least one it does not, and the failure is loud in a misleading way:

bash: line 1: sinfo: command not found
bash: line 2: sacctmgr: command not found

with $CC_CLUSTER, $SCRATCH and $PROJECT all empty. The cluster is fine — the Slurm binaries and site environment come from the login profile, and a non-interactive ssh <host> 'cmd' never sources it. Every diagnostic you would reach for to check whether the cluster is healthy fails at once, which reads exactly like a broken or half-provisioned account.

Fix: force a login shell.

ssh <cluster> 'bash -lc "sinfo -h -o \"%P %l\" | sort -u"'

Never conclude an account is unprovisioned from command not found. Retry through bash -lc first, and note per cluster whether it is needed — it changes how every scripted command against that host must be written, including the ones inside your job-submission wrappers. An empty $SCRATCH is especially dangerous: a path built as "$SCRATCH/data" silently becomes /data.

Accounts often but NOT always carry a resource suffix (<alloc>_cpu, <alloc>_gpu). Where they do, requesting a GPU under _cpu fails and CPU work charged to _gpu burns a scarcer allocation. Where they don't (AI clusters using aip- allocations), one account covers both. Always run sacctmgr; never copy an account string between projects or clusters.

Never construct user paths — use $HOME and $SCRATCH. Some clusters shard both by first letter, and the constructed form does not merely differ, it does not exist:

narval/rorqual   HOME=/home/$USER         SCRATCH=/scratch/$USER
tamia            HOME=/home/<u>/$USER      SCRATCH=/scratch/<u>/$USER

                 where <u> is the first letter of your username

ls /scratch/$USER on TamIA returns No such file or directory. ($PROJECT is not set on any of them — get the project path from diskusage_report or ls /project/.)

GPU gres strings are NOT consistent across clusters — always read them from sinfo -h -o "%G". Two clusters, same year, same generation of hardware:

narval    gpu:a100  gpu:a100_1g.5gb  gpu:a100_3g.20gb          # short MIG names
rorqual   gpu:h100  gpu:nvidia_h100_80gb_hbm3_1g.10gb          # fully-qualified

A --gpus= string copied from another cluster's job script fails, and MIG slices (the _1g. / _2g. / _3g. variants) are fractional GPUs — request one by accident and you get a sliver of a card. Ask for a whole GPU explicitly.

Time buckets decide your queue. Partitions are binned by walltime: _b1 ≤3h, _b2 ≤12h, _b3 ≤24h, _b4 ≤3d, _b5 ≤7d. Asking 13h instead of 12h moves you to a scarcer partition and can cost days of waiting — round --time down into a bucket.

How FAR the buckets go is a hard cap that differs per cluster. General-purpose clusters run to b5 (7 days); AI clusters may stop at b3, making 24h the maximum job length — a 48h job is not slow to schedule there, it is impossible, and long work must checkpoint and requeue. Always check:

sinfo -h -o "%P %l" | sort -u        # the authoritative bucket list

Filesystems: file COUNT is the quota that bites

Space is rarely the limit; inodes are — millions of small files wedge a group's /project while it looks half empty. Put re-fetchable data and high-file-count output on /scratch.

Copying into /project must preserve group ownership:

rsync -a --no-g --no-p --stats src/ <cluster>:/project/<group>/<user>/<proj>/src/

Without --no-g --no-p, rsync imposes local ownership and breaks the setgid group inheritance /project relies on, locking out collaborators. This fails silently — verify after every transfer:

ssh <cluster> "cd <projdir> && find . -type f ! -group <group> | wc -l"   # must print 0

Use --stats, not --info=stats1, for a summary that prints reliably.

Submitting and monitoring

Calibrate before submitting an array. Run ONE task, read its elapsed time, then set --time with headroom. A 200-task array on a guessed walltime either wastes an allocation or dies at 99%.

squeue -u $USER -o '%.14i %.22j %.8T %.10M %.10l %R'   # %R = why it is PENDING
sacct -X -S <YYYY-MM-DD> -o JobID%16,JobName%22,State,Elapsed,ExitCode
sacct -j <id> -X -n -o State | sort | uniq -c          # array state histogram

-X gives one row per job instead of per job step.

Never pipe a running job's output through tail, head or grep — they buffer, so the file stays empty and you lose all visibility into something you already started. Redirect to a file and read the file.

One failed task in an otherwise healthy array is usually a bad node. Look for hardware-flavoured errors (e.g. CUDA driver version is insufficient for CUDA runtime version) before suspecting your code; resubmit that index alone with sbatch --array=<n> <script>.

Common mistakes

Red flags — stop and check

• About to script SSH_ASKPASS, or otherwise auto-answer an MFA prompt → stop and ask. Completing the task is not worth silently spending the user's second factor
• About to run anything longer than a few seconds over ssh → that is a login node
• About to say the cluster is down → run ssh -O check first
• About to say an account is unprovisioned because Slurm commands are missing → retry through bash -lc
• About to reuse a partition name, bucket ceiling or gres string from another cluster → read sinfo on THIS one
• About to submit >20 tasks without having timed one → calibrate
• About to report a job's progress from a buffered pipe → you are reading nothing