Digital Research Alliance

Digital Research Alliance of Canada (DRAC)

Overview

DRAC (alliancecan.ca, formerly Compute Canada) runs Canada's national HPC clusters. All use Slurm — so for sbatch/monitoring/throttling/GPU-fraction mechanics, use the slurm skill. This skill is the DRAC-specific layer: which cluster, which account, where the docs are, and how to run across several clusters at once.

Core principle: you have one set of allocations usable on several clusters; spread your jobs across whichever cluster is least busy, and let logs/accounting tell you the outcome.

Where the documentation is

Note: the wiki is behind an anti-bot wall (Anubis) — fetch it from a real browser or, on a login node, the info is also in sinfo/sacctmgr.

Where the GPUs are (2026) — pick the least-busy one

DRAC migrated to new systems in 2024–2025. Retired: Cedar → Fir, Beluga → Rorqual, Graham → Nibi, Niagara → Trillium. Mist retired.

Most GPUs / biggest H100 pools: Fir, Rorqual, Nibi. For pure AI workloads the Killarney/TamIA/Vulcan AI clusters often have the shortest queues (they need a separate AIP allocation — see below). See references/clusters.md for the full table and references/multi-cluster.md for the spread-the-load strategy.

Checking availability before you submit

DRAC has no global "free GPU" dashboard — check per cluster from its login node:

sinfo -p gpubase_bygpu_b1 -t idle,mix -o "%n %t %C %G"   # free GPU nodes + types (partition names vary)
squeue -p gpu -t PENDING | wc -l                          # how deep is the queue
sshare -U                                                  # your fairshare (low = you'll wait)
partition-stats                                            # alliance helper, if present

Submit to the cluster with idle/mix GPU nodes and a short pending queue. When in doubt, submit a short version to two clusters and keep whichever starts first.

Accounts & allocations (the --account flag)

Every job needs --account=<group>. Format tells you the priority tier:

sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES  # what you can charge to

def- jobs run when there's room; rrg- jobs get scheduled ahead within their share. Use your rrg- account for the work that matters, def- to soak up idle.

Lua auto-routing: usually OMIT --partition

DRAC is not vanilla Slurm. A custom Lua submission plugin auto-routes your job to a partition based on --time. So, opposite to generic Slurm advice:

• Omit --partition unless you have a concrete reason (e.g. forcing CPU-only). Setting it with a mismatched --time → instant rejection (partition does not exist or cannot fit).
• Walltime tiers go up to 7 days; over-7-day requests are rejected.
• srun/salloc always route to the interactive partition.
• Defaults are hostile: --time defaults to 60 min and --mem defaults to a tiny cgroup-enforced value. Always set both, or jobs die at 60 min / OOM in seconds.
• Job scripts must start with #!/bin/bash (the Lua plugin checks this).

DRAC GPU request syntax — discover, never guess

GPU type names differ by cluster/node, and bare --gres=gpu:N is rejected on the general-purpose clusters (Fir/Nibi/Rorqual) — always use the full type string. Discover the exact strings first:

sinfo -h -o "%G" --Node | sort -u            # all GRES; MIG/shard entries appear here
sinfo -h -o "%N %G %f" --Node | sort -u      # + features column (e.g. softmig)
scontrol show node <node> | grep -E 'Gres=|CfgTRES='   # full detail for one node

#SBATCH --gres=gpu:h100:1             # typed (h100 / a100 / v100l / l40s) — use exact string
#SBATCH --gres=gpu:h100:4             # multiple (verify node topology first)

Fractions — two mechanisms, both count=1:

# 1) Hardware MIG slice (A100/H100). Slice GRES shows in sinfo, e.g. gpu:a100.1g.10gb:7
#SBATCH --gres=gpu:a100.1g.10gb:1     # ~1/7, 10GB   (also a100.2g.20gb, a100.3g.40gb, ...)

# 2) Soft-MIG / shard (e.g. some L40S nodes). Node shows composite: gpu:l40s:4,shard:l40s:16
#SBATCH --gres=gpu:l40s.4:1           # 1/4 slice via dot-denominator notation

Slice-name spelling is exact and varies (dot form a100.1g.10gb on some clusters, underscore a100_1g.5gb in older docs) — copy the literal string from sinfo, don't type it from memory. Only request a shard fraction on nodes whose %G shows a shard: entry.

Whole-node vs single-GPU: clusters bundle CPU/mem per GPU. Asking for 1 GPU with disproportionate cores/mem queues forever. Match the node ratio (e.g. Trillium GPU node = 4 GPUs → single-GPU job gets ~1/4 of the node's CPU/mem).

Compute nodes have no internet (proxy, caches, TMPDIR)

The prolog injects environment that trips up naive jobs:

• No direct internet on compute nodes — HTTP/S routes through a squid proxy (http_proxy=http://squid:3128). Use https:// git URLs; SSH-git and raw TCP may be blocked. Login nodes have full internet.
• $SLURM_TMPDIR (fast node-local disk) is wiped at job end — use it for unpacking datasets, not for keeps. Don't hardcode /tmp (path varies by cluster).
• The prolog points XDG_CACHE_HOME at $SLURM_TMPDIR/cache, so model/dataset downloads vanish. In any job that downloads, redirect caches to $SCRATCH:

  export HF_HOME=$SCRATCH/hf_cache XDG_CACHE_HOME=$SCRATCH/cache
  

Software: modules & the wheelhouse (CVMFS/Lmod)

Software comes from CVMFS via Lmod (module), not from apt/system Python, and compute nodes can't pip install from PyPI (no internet). Essentials:

• Discover with module spider <name> (NOT module avail — it hides higher tiers). module spider <name>/<ver> prints the exact prerequisite load line.
• Load a stack: module --force purge; module load StdEnv/2023 gcc/<v> cuda/<v> python/<v>.
• Build venvs from the wheelhouse: avail_wheels <pkg> then pip install --no-index <pkg> (drop --no-index only if a wheel is missing).
• Re-load modules inside every job script (jobs don't inherit your login shell).
• diskusage_report for real quota/inode usage.

Full details in references/cvmfs-lmod.md.

Storage & quotas (the gotchas that kill jobs)

• A full quota or inode cap makes jobs die with confusing I/O errors — check with diskusage_report (alliance helper) before big runs.
• Datasets = many small files blow the inode cap; pack into a few archives / use $SLURM_TMPDIR (fast node-local disk) for unpacking at job start.
• Don't sbatch from /home on the GP clusters (Fir/Nibi/Rorqual) — submission is rejected. cd $SCRATCH/... (or $PROJECT) and submit from there.

Running across multiple clusters

This is DRAC's superpower: your allocation works on several clusters, so a sweep can fan out across all of them. See references/multi-cluster.md for the full playbook (one repo on each cluster via git, per-cluster manifests, agent loop that polls every cluster and rebalances). Short version:

1. Same git repo cloned on each cluster; account + partition are the only per-site diffs.
2. Submit the sweep split across clusters (or fully to whichever is idle).
3. An agent loop (ScheduleWakeup / loop skill) SSHes each cluster, runs squeue --me/sacct, cancels revealed-answer jobs, refills the idle cluster.
4. Cost model: def- = free/opportunistic everywhere; rrg-/aip- consume your competed allocation — spend it on the cluster where it buys the most idle GPUs, soak the rest with def-.

Common mistakes

Credits

DRAC-specific operational details (Lua auto-routing, GRES/MIG/shard discovery, the squid-proxy / no-internet-on-compute-nodes behavior, and the CVMFS/Lmod wheelhouse workflow) were learned from and verified against ualberta-rcg/drac-agent-skills (MIT, © University of Alberta Research Computing Group) — a DRAC-only skill set with a live docs RAG API and deeper CVMFS/Apptainer coverage. Use it alongside this skill if you work exclusively on DRAC.