# Digital Research Alliance

> Use when working on the Digital Research Alliance of Canada (DRAC / alliancecan / formerly Compute Canada) HPC clusters — Fir, Rorqual, Nibi, Trillium, Narval, and the AI clusters Killarney, TamIA, Vulcan. Covers where the docs are, where the GPUs are and how busy they are, accounts/allocations (def-/rrg-/rpp-/AIP), the alliance gres/MIG syntax, storage quotas, and running across multiple clusters at once.

- Skill: `domenicrosati/digital-research-alliance` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add domenicrosati/digital-research-alliance`
- Raw SKILL.md: https://api.skillmd.com/api/skills/domenicrosati/digital-research-alliance/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: domenicrosati (https://skillmd.com/u/domenicrosati)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/domenicrosati/digital-research-alliance

---


# Digital Research Alliance of Canada (DRAC)

## Overview

DRAC (alliancecan.ca, formerly Compute Canada) runs Canada's national HPC clusters.
All use **Slurm** — so for sbatch/monitoring/throttling/GPU-fraction mechanics,
use the `slurm` skill. This skill is the DRAC-specific layer: which cluster, which
account, where the docs are, and how to run across several clusters at once.

Core principle: **you have one set of allocations usable on several clusters;
spread your jobs across whichever cluster is least busy, and let logs/accounting
tell you the outcome.**

## Where the documentation is

| Resource | URL | For |
|---|---|---|
| Wiki (the docs) | https://docs.alliancecan.ca | everything — start here |
| Status / outages | https://status.alliancecan.ca | is the cluster up / in maintenance |
| CCDB portal | https://ccdb.alliancecan.ca | accounts, allocations, usage, add users |
| Allocation (RAC) | https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources | apply for priority allocation |
| Running jobs | https://docs.alliancecan.ca/wiki/Running_jobs | sbatch on alliance |
| GPUs with Slurm | https://docs.alliancecan.ca/wiki/Using_GPUs_with_Slurm | gres/MIG syntax (authoritative) |
| Per-cluster pages | https://docs.alliancecan.ca/wiki/<Cluster> (e.g. `/wiki/Narval`) | node/GPU/partition specifics |

Note: the wiki is behind an anti-bot wall (Anubis) — fetch it from a real browser
or, on a login node, the info is also in `sinfo`/`sacctmgr`.

## Where the GPUs are (2026) — pick the least-busy one

DRAC migrated to new systems in 2024–2025. **Retired:** Cedar → Fir, Beluga →
Rorqual, Graham → Nibi, Niagara → Trillium. **Mist** retired.

| Cluster | Site | GPUs | Notes |
|---|---|---|---|
| **Nibi** | Waterloo | ~288× **H100-80G** | general purpose, ~134k CPU cores |
| **Rorqual** | Calcul Québec | ~324× **H100-80G** | general purpose, ~137k CPU cores |
| **Fir** | SFU | **H100-80G** (largest GPU pool) | general purpose (replaces Cedar) |
| **Trillium** | Toronto/SciNet | **H100** GPU subcluster | GPU node = 4 GPUs; single-GPU = 1/4 node |
| **Narval** | Calcul Québec | **A100-40G** (MIG-capable) | older but still big; use MIG to save GPUs |
| **Killarney** | Vector Inst. (Ontario) | **H100 + L40S-48G** | **AI-dedicated** (PAICE) |
| **TamIA** | Université Laval | **H100 + H200** | **AI-dedicated** (PAICE) |
| **Vulcan** | Amii (Alberta) | **L40S-48G** | **AI-dedicated** (PAICE); has on-cluster inference (Kubeflow) |
| **Arbutus** | Victoria | cloud GPUs | **OpenStack cloud, NOT Slurm** — VMs, not sbatch |

**Most GPUs / biggest H100 pools: Fir, Rorqual, Nibi.** For pure AI workloads the
**Killarney/TamIA/Vulcan** AI clusters often have the shortest queues (they need a
separate AIP allocation — see below). See `references/clusters.md` for the full
table and `references/multi-cluster.md` for the spread-the-load strategy.

## Checking availability before you submit

DRAC has no global "free GPU" dashboard — check per cluster from its login node:

```bash
sinfo -p gpubase_bygpu_b1 -t idle,mix -o "%n %t %C %G"   # free GPU nodes + types (partition names vary)
squeue -p gpu -t PENDING | wc -l                          # how deep is the queue
sshare -U                                                  # your fairshare (low = you'll wait)
partition-stats                                            # alliance helper, if present
```
Submit to the cluster with idle/mix GPU nodes and a short pending queue. When in
doubt, submit a short version to two clusters and keep whichever starts first.

## Accounts & allocations (the `--account` flag)

Every job needs `--account=<group>`. Format tells you the priority tier:

| Prefix | Tier | Source |
|---|---|---|
| `def-<pi>` | default / opportunistic | every PI gets one automatically |
| `rrg-<pi>` / `rpp-<pi>` | **priority** allocation | won via the annual Resource Allocation Competition (RAC) |
| `aip-<pi>` / AIP | **AI clusters** (Killarney/TamIA/Vulcan) | separate AI allocation program |

```bash
sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES  # what you can charge to
```
`def-` jobs run when there's room; `rrg-` jobs get scheduled ahead within their
share. Use your `rrg-` account for the work that matters, `def-` to soak up idle.

## Lua auto-routing: usually OMIT `--partition`

DRAC is **not** vanilla Slurm. A custom **Lua submission plugin auto-routes your
job to a partition based on `--time`**. So, opposite to generic Slurm advice:

- **Omit `--partition`** unless you have a concrete reason (e.g. forcing CPU-only).
  Setting it with a mismatched `--time` → instant rejection
  (`partition does not exist or cannot fit`).
- Walltime tiers go up to **7 days**; over-7-day requests are rejected.
- `srun`/`salloc` always route to the *interactive* partition.
- **Defaults are hostile:** `--time` defaults to **60 min** and `--mem` defaults to
  a tiny cgroup-enforced value. Always set both, or jobs die at 60 min / `OOM` in
  seconds.
- Job scripts must start with `#!/bin/bash` (the Lua plugin checks this).

## DRAC GPU request syntax — discover, never guess

GPU type names differ by cluster/node, and **bare `--gres=gpu:N` is rejected on the
general-purpose clusters (Fir/Nibi/Rorqual)** — always use the full type string.
Discover the exact strings first:

```bash
sinfo -h -o "%G" --Node | sort -u            # all GRES; MIG/shard entries appear here
sinfo -h -o "%N %G %f" --Node | sort -u      # + features column (e.g. softmig)
scontrol show node <node> | grep -E 'Gres=|CfgTRES='   # full detail for one node
```

```bash
#SBATCH --gres=gpu:h100:1             # typed (h100 / a100 / v100l / l40s) — use exact string
#SBATCH --gres=gpu:h100:4             # multiple (verify node topology first)
```

**Fractions — two mechanisms, both `count=1`:**

```bash
# 1) Hardware MIG slice (A100/H100). Slice GRES shows in sinfo, e.g. gpu:a100.1g.10gb:7
#SBATCH --gres=gpu:a100.1g.10gb:1     # ~1/7, 10GB   (also a100.2g.20gb, a100.3g.40gb, ...)

# 2) Soft-MIG / shard (e.g. some L40S nodes). Node shows composite: gpu:l40s:4,shard:l40s:16
#SBATCH --gres=gpu:l40s.4:1           # 1/4 slice via dot-denominator notation
```
Slice-name spelling is exact and varies (dot form `a100.1g.10gb` on some clusters,
underscore `a100_1g.5gb` in older docs) — **copy the literal string from `sinfo`**,
don't type it from memory. Only request a shard fraction on nodes whose `%G` shows a
`shard:` entry.

**Whole-node vs single-GPU:** clusters bundle CPU/mem per GPU. Asking for 1 GPU
with disproportionate cores/mem queues forever. Match the node ratio (e.g.
Trillium GPU node = 4 GPUs → single-GPU job gets ~1/4 of the node's CPU/mem).

## Compute nodes have no internet (proxy, caches, TMPDIR)

The prolog injects environment that trips up naive jobs:

- **No direct internet on compute nodes** — HTTP/S routes through a squid proxy
  (`http_proxy=http://squid:3128`). Use **`https://` git URLs**; SSH-git and raw TCP
  may be blocked. Login nodes have full internet.
- `$SLURM_TMPDIR` (fast node-local disk) is **wiped at job end** — use it for
  unpacking datasets, not for keeps. Don't hardcode `/tmp` (path varies by cluster).
- The prolog points `XDG_CACHE_HOME` at `$SLURM_TMPDIR/cache`, so model/dataset
  downloads vanish. In any job that downloads, redirect caches to `$SCRATCH`:
  ```bash
  export HF_HOME=$SCRATCH/hf_cache XDG_CACHE_HOME=$SCRATCH/cache
  ```

## Software: modules & the wheelhouse (CVMFS/Lmod)

Software comes from **CVMFS** via **Lmod** (`module`), not from `apt`/system Python,
and **compute nodes can't `pip install` from PyPI** (no internet). Essentials:

- Discover with `module spider <name>` (NOT `module avail` — it hides higher tiers).
  `module spider <name>/<ver>` prints the exact prerequisite load line.
- Load a stack: `module --force purge; module load StdEnv/2023 gcc/<v> cuda/<v> python/<v>`.
- Build venvs from the **wheelhouse**: `avail_wheels <pkg>` then
  `pip install --no-index <pkg>` (drop `--no-index` only if a wheel is missing).
- Re-load modules **inside** every job script (jobs don't inherit your login shell).
- `diskusage_report` for real quota/inode usage.

Full details in `references/cvmfs-lmod.md`.

## Storage & quotas (the gotchas that kill jobs)

| Space | Backed up | Use for | Watch |
|---|---|---|---|
| `$HOME` | yes | code, small | small quota; not for job I/O |
| `$SCRATCH` | **no, purged** | large job I/O, datasets | files **auto-deleted** after ~60 days |
| `$PROJECT` | yes | shared datasets, results | **per-group quota AND inode (file-count) cap** |

- A full quota or inode cap makes jobs die with confusing I/O errors — check with
  `diskusage_report` (alliance helper) **before** big runs.
- Datasets = many small files blow the inode cap; pack into a few archives / use
  `$SLURM_TMPDIR` (fast node-local disk) for unpacking at job start.
- **Don't `sbatch` from `/home` on the GP clusters (Fir/Nibi/Rorqual)** — submission
  is rejected. `cd $SCRATCH/...` (or `$PROJECT`) and submit from there.

## Running across multiple clusters

This is DRAC's superpower: your allocation works on several clusters, so a sweep
can fan out across all of them. See `references/multi-cluster.md` for the full
playbook (one repo on each cluster via git, per-cluster manifests, agent loop that
polls every cluster and rebalances). Short version:

1. Same git repo cloned on each cluster; account + partition are the only per-site diffs.
2. Submit the sweep split across clusters (or fully to whichever is idle).
3. An agent loop (`ScheduleWakeup` / `loop` skill) SSHes each cluster, runs
   `squeue --me`/`sacct`, cancels revealed-answer jobs, refills the idle cluster.
4. **Cost model:** `def-` = free/opportunistic everywhere; `rrg-`/`aip-` consume
   your competed allocation — spend it on the cluster where it buys the most idle
   GPUs, soak the rest with `def-`.

## Common mistakes

| Mistake | Fix |
|---|---|
| Submitting to Arbutus with sbatch | Arbutus is OpenStack cloud, not Slurm — use VMs |
| `pip install` from PyPI on compute node | no internet on compute nodes; use `--no-index` wheelhouse on login node |
| Big dataset of small files in `$PROJECT` | blows inode cap; archive + unpack to `$SLURM_TMPDIR` |
| Relying on `$SCRATCH` for keeps | it's purged (~60 days) — move results to `$PROJECT` |
| AI-cluster job with a `def-`/`rrg-` account | Killarney/TamIA/Vulcan need an AIP allocation |
| One cluster, long queue | spread across Fir/Rorqual/Nibi + AI clusters (multi-cluster.md) |
| Whole A100 for a tiny job on Narval | request a MIG slice (discover string, e.g. `a100.3g.40gb`) |
| Setting `--partition` | usually omit it — Lua auto-routes on `--time` |
| `pip install` on a compute node | no internet there — use the wheelhouse / install on login node |
| Job dies at exactly 60 min | `--time` defaulted — always set it |

## Credits

DRAC-specific operational details (Lua auto-routing, GRES/MIG/shard discovery, the
squid-proxy / no-internet-on-compute-nodes behavior, and the CVMFS/Lmod wheelhouse
workflow) were learned from and verified against
[**ualberta-rcg/drac-agent-skills**](https://github.com/ualberta-rcg/drac-agent-skills)
(MIT, © University of Alberta Research Computing Group) — a DRAC-only skill set with
a live docs RAG API and deeper CVMFS/Apptainer coverage. Use it alongside this skill
if you work exclusively on DRAC.