Hpc

Musser Lab HPC Reference

Yale Center for Research Computing (YCRC) cluster conventions for the Musser Lab. Full YCRC documentation: https://docs.ycrc.yale.edu/

1. Getting Started

Account setup

Request an account at https://research.computing.yale.edu/account-request. You need a Yale NetID and PI approval (Jacob Musser).

SSH access

ssh <netid>@mccleary.ycrc.yale.edu
ssh <netid>@bouchet.ycrc.yale.edu
ssh <netid>@misha.ycrc.yale.edu

Use SSH keys for passwordless access. Add to ~/.ssh/config:

Host mccleary
    HostName mccleary.ycrc.yale.edu
    User <netid>

Host bouchet
    HostName bouchet.ycrc.yale.edu
    User <netid>

Host misha
    HostName misha.ycrc.yale.edu
    User <netid>

Note on "passwordless": SSH keys get you past the password, but YCRC still requires Duo 2FA (a keyboard-interactive passcode) on every login-node connection. For interactive terminal use that's just one extra prompt. For Positron / VS Code Remote-SSH, the non-interactive ProxyCommand can't answer Duo — you must use SSH ControlMaster multiplexing so one interactive login is reused. See references/positron-ssh-setup.md for the full template and setup order — it covers two tiers: a basic session, and a persistent one that survives VPN/laptop disconnects (the allocation is held in tmux on the login node, so a dropped connection no longer kills it).

First job

After logging in, test with a minimal job:

sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --partition=devel
#SBATCH --time=0:05:00
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
echo "Hello from $(hostname) at $(date)"
EOF

Check status with squeue --me, view output in slurm-<jobid>.out.

2. Cluster Overview

Which cluster to use

• YCGA sequencing data analysis → McCleary (while available) — the ycga partition is exempt from compute charges
• GPU jobs (training, PROST structure search) → Bouchet (H200, RTX Pro 6000 Blackwell, RTX 5000 Ada)
• General compute (phylogenetics, alignment, mapping) → McCleary or Bouchet
• Wu Tsai affiliated work → Misha

Open OnDemand (web portal)

Each cluster has a web portal for Jupyter, RStudio, VSCode, and Remote Desktop:

• McCleary: https://ood-mccleary.ycrc.yale.edu
• Bouchet: https://ood-bouchet.ycrc.yale.edu
• Misha: https://ood-misha.ycrc.yale.edu

Max 4 interactive app instances per user simultaneously. Yale VPN required off-campus.

Login node policy

Never run heavy computation on login nodes. Acceptable login-node activities:

• Snakemake orchestration (dispatching jobs to SLURM)
• Git operations
• Conda/mamba environment management
• File inspection, editing scripts
• Small interactive commands (wc, head, ls, etc.)

Everything else must be submitted as a SLURM job.

Transfer nodes

Transfer nodes (e.g., transfer1.bouchet) are for data transfer only (rsync, Globus, scp). Do not use them to run tools — they often have older CPUs that cause "Illegal instruction" errors with compiled binaries. If you need to run module load, prefetch, fasterq-dump, or any analysis tool, use an interactive compute node instead (see Interactive jobs below).

3. Storage

Lab storage paths

McCleary has TWO PI storage allocations of very different size — pick deliberately:

• /vast/palmer/pi/musser — large (~20 TiB), Palmer VAST. This is canonical for raw data, shared data folders, and anything bulky.
• /gpfs/gibbs/project/musser — small (~1 TiB), Gibbs. Easy to fill; some lab members land here by default (e.g. older dc2423/ working dirs), but it is NOT where new bulk data should go. Prefer Palmer for raw data / shared folders unless a tool specifically needs Gibbs.

When in doubt, check both with getquota and put bulk data on whichever has headroom (usually Palmer).

Shared lab data folder (raw sequencing data, cross-cluster)

Raw sequencing deliveries shared across projects/people live in a shared data folder under PI storage. The intermediate folder name differs between clusters — Bouchet uses shared/, McCleary uses Data/ — so it is NOT a clean root-prefix swap; use the per-cluster path below:

Under either root the internal layout matches, e.g. …/YCGA_DATA_POSTED/PACBIO/... and …/YCGA_DATA_POSTED/SingleCell/.... On Bouchet, ~/project_pi_jm284/jm284/Data/YCGA_DATA_POSTED is a symlink into shared/YCGA_DATA_POSTED. Feed these from the lab NAS via Globus. In scripts, parameterize the root (note the differing folder name per cluster):

SHARED_DATA="${SHARED_DATA:-$HOME/project_pi_jm284/shared}"   # Bouchet; McCleary: /vast/palmer/pi/musser/Data

On Bouchet, lab members typically work via the home symlink ~/project_pi_jm284/ and keep their projects under a per-user subdirectory: ~/project_pi_jm284/<netid>/projects/<project_name>/. This resolves to /nfs/roberts/project/pi_jm284/<netid>/projects/<project_name>/ underneath. Either form works; the home-symlink form is more discoverable and matches cd history.

Shared lab databases on Bouchet

Reference databases that are reusable across projects (BUSCO lineages, eggNOG DB, DIAMOND-formatted UniProt, sponge mitochondrial reference sets, etc.) live at:

~/project_pi_jm284/shared/databases/
  busco_lineages/      # BUSCO odb10 datasets (metazoa, eukaryota, etc.)
  ...                  # other shared DBs as they're added

Resolves to /nfs/roberts/project/pi_jm284/shared/databases/ under the symlink.

When writing batch or setup scripts that reference these shared DBs, use a parameter that defaults to this location and can be overridden via env var:

SHARED_DBS="${SHARED_DBS:-$HOME/project_pi_jm284/shared/databases}"

This way the script picks up the lab default automatically but can be redirected on a non-Bouchet machine, or to a different location for testing, without editing.

Storage policies

Important:

• Never store conda environments on scratch — they will be purged after 60 days
• Never store raw data only on scratch — keep originals in PI storage
• You will receive email notification one week before scratch files are purged
• Do not artificially modify file timestamps to circumvent the purge policy
• Check quotas: getquota (McCleary) | List paths: mydirectories

Project organization on the cluster

Mirror the local project structure in PI storage or project space:

/nfs/roberts/project/pi_jm284/<project_name>/
  .git/              # Same repo as local — sync via git push/pull
  .claude/           # Project docs, plans, findings
  batch/             # SLURM batch scripts (tracked in git)
  logs/              # SLURM output files (tracked in git)
  scripts/           # Analysis scripts
  data/              # External/immutable inputs (gitignored, sync via Globus)
  outs/              # Script-produced outputs (gitignored)
  environment.yml    # Conda environment specification

Use scratch only for large temporary files (sort buffers, intermediate alignments) that can be regenerated. Never store the project itself on scratch — use PI storage.

Dual-environment projects (local + cluster)

When a project is worked on both locally and on the cluster, the same git repo is cloned in both places. Conventions:

• Git syncs scripts, docs, logs, and batch files. Always git pull before starting work in either environment.
• Data syncs via Globus. Large files (data/, tool outputs) are gitignored and transferred manually. Not all data exists in both places.
• Batch scripts use BASEDIR=$(git rev-parse --show-toplevel) — no hardcoded paths. This makes scripts work regardless of where the repo is cloned.
• Logs are tracked in git. The cluster session commits logs after jobs complete; pull locally to review results.
• Cluster-only directories (e.g., cellranger_refs/, raw FASTQ staging directories) are added to .gitignore on a per-project basis.
• The project CLAUDE.md documents the dual-environment setup, including the cluster path and which directories are cluster-only.

4. SLURM Job Scheduling

Full docs: https://docs.ycrc.yale.edu/clusters-at-yale/job-scheduling/

Lab default batch script template

Every batch script starts from these defaults. Override per-job as needed.

#!/bin/bash
#SBATCH --job-name=<tool>_<brief_description>
#SBATCH --partition=day
#SBATCH --time=4:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=5G
#SBATCH --output=logs/slurm-%j.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=<your email>

# ── Provenance ────────────────────────────────────────
BASEDIR=$(git rev-parse --show-toplevel)
cd "$BASEDIR"

echo "=== PROVENANCE ==="
echo "Job ID:      $SLURM_JOB_ID"
echo "Script:      \$0"
echo "Git hash:    $(git rev-parse HEAD)"
echo "Git dirty:   $(git status --porcelain | head -5)"
echo "Date:        $(date -Iseconds)"
echo "Node:        $(hostname)"

# Modules (cluster-only tools — always pin versions)
module purge
module load Tool/1.2.3

module list 2>&1

# Conda (project environment)
module load miniconda
conda activate myenv
echo "Conda env:   $CONDA_DEFAULT_ENV"

# Log versions of key tools used in this script
echo "tool:        $(tool --version | head -1)"
echo "=== END PROVENANCE ==="
# ──────────────────────────────────────────────────────

# ── Main ──────────────────────────────────────────────
# Your commands here

The provenance block prints to stdout, which SLURM captures in the log file. Since logs are tracked in git, the full record (script version, tool versions, environment state) is version-controlled alongside the code that produced the results.

Provenance block requirements:

• BASEDIR + cd — ensures git commands and relative paths work
• git rev-parse HEAD — which version of the code ran
• git status --porcelain — catches uncommitted changes (scripts called by the batch job are covered by the git hash only if the tree is clean)
• module list — all loaded modules and their exact versions
• conda activate + env name — which conda environment was used
• Tool version lines — one per tool actually invoked in the script

File organization:

• Batch scripts → batch/ subdirectory
• Log files → logs/ subdirectory (create with mkdir -p logs before first submit)
• batch/ is tracked in git (scripts are code)
• logs/ is tracked in git (logs are the reproducibility record — commit after jobs complete so provenance is preserved)

Critical: No space between # and SBATCH — otherwise the directive is ignored.

Common directives

Notification options

For array jobs, always include ARRAY_TASKS. Without it, individual task failures within a running array don't trigger emails — you only find out when the whole array finishes.

GPU jobs

GPUs must be explicitly requested with --gpus. Key GPU partitions:

For GPU jobs, also module load CUDA before conda activation.

For choosing the least-congested GPU partition before submitting, and for switching a pending job's partition to a faster queue, see references/gpu-partition-tactics.md.

Interactive jobs

Interactive work belongs in devel. YCRC makes devel the default partition for salloc, and interactive jobs are normally permitted on devel (or gpu_devel for a GPU) or on a private partition you have been explicitly authorized to use — not generally on the other public partitions. Do not reach for day just because an interactive job wants more time or more cores; that is a signal to write a batch script.

IDE sessions — Positron or VS Code, including Claude Code running inside them — are covered by a stricter, explicit rule. YCRC states that VS Code jobs found outside the devel partitions may be terminated without notice; this skill treats Positron Remote-SSH as the same kind of IDE workload, so the same rule is assumed to apply to it.

# Quick interactive session (testing, short tasks)
salloc -p devel -c 4 --mem=16G -t 2:00:00

# Claude Code / Positron working session (writing scripts, running them, submitting batch jobs)
salloc -p devel -c 4 --mem=32G -t 6:00:00 --job-name=positron-devel

Sizing an interactive session: 4 CPUs, up to 32 GB RAM. Most interactive work (writing scripts, parsing TSVs/GFFs, pandas, matplotlib) needs <4 GB; the extra memory is headroom for occasional spikes (loading large SQLite DBs, sorting big BLAST outputs, multi-panel figures).

devel per-user limits are aggregate across all of your devel jobs, not per job (see references/partitions.md):

A single 4-CPU session therefore consumes your entire devel CPU allowance on either cluster, and McCleary permits only one submitted devel job at a time.

Heavy compute (DIAMOND, Cell Ranger, STAR, BRAKER, eggNOG-mapper, PROST) should always be submitted as batch jobs — never run on the interactive node. Rule of thumb: if it takes

5 minutes or needs >4 CPUs, write a batch script.

Add --x11 for graphical forwarding (requires X11 setup).

For setting up Positron (VS Code) to connect to an interactive session via SSH, see references/positron-ssh-setup.md. It documents two tiers — a basic session (quick to set up; a laptop/VPN drop kills the allocation) and a persistent session (a bit more setup; the allocation survives disconnects via a tmux-held salloc + helper scripts, with reconnect instructions). The persistent tier buys no extra time and no extra resources — it is the same 6-hour devel job inside the same caps. What it buys is that a laptop sleep, VPN drop, or closed lid no longer forfeits the rest of that session and sends you back to the queue.

Job monitoring

Resource efficiency

Always check resource usage with jobstats after jobs complete. Request only what you need — wasteful allocations slow scheduling for everyone.

Job arrays

For many similar jobs (e.g., processing multiple samples), use job arrays or Dead Simple Queue (dsq) rather than submitting hundreds of individual jobs. Rate limit: 200 job submissions per hour.

#!/bin/bash
#SBATCH --array=1-50
#SBATCH --partition=day
#SBATCH --time=2:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4G
#SBATCH --output=logs/slurm-%A_%a.out
#SBATCH --mail-type=BEGIN,END,FAIL,ARRAY_TASKS
#SBATCH --mail-user=<your email>

SAMPLE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)
# Process $SAMPLE

5. Partition Quick-Reference

See references/partitions.md for full partition tables (McCleary, Bouchet, Misha) with time limits, per-user resource limits, GPU types, and node counts.

Quick summary for partition selection:

• General compute → day (1-day limit, generous CPU/memory)
• Long jobs → week (7-day) or long (28-day, McCleary only)
• GPU → gpu (RTX 5000 Ada on Bouchet), gpu_h200 (H200), gpu_rtx6000 (RTX Pro 6000 Blackwell)
• Interactive/testing → devel (6-hour limit, strict per-user caps)
• YCGA data → ycga on McCleary (exempt from compute charges)
• Big memory → bigmem (up to 4 TiB/node on Bouchet)
• Preemptable → scavenge (free idle resources, may be killed)

6. Tool Resource Profiles

See references/tool_profiles.md for SLURM resource recommendations per bioinformatics tool (CPUs, memory, time, partition, GPU). Always check jobstats after initial runs and adjust.

7. Environment Management

Modules vs conda: the hybrid rule

For any given tool, use one or the other — never both. If a tool is available via both module and conda, choose one and stick with it. Having the same tool in both creates PATH conflicts where the activation order silently determines which version runs.

Always pin module versions — use module load CellRanger/9.0.1, never bare module load CellRanger. Unpinned modules silently change when the cluster updates defaults. The provenance block in the batch template logs loaded versions, but pinning prevents the drift in the first place.

Conda (Python + bioinformatics tools)

module purge
module load miniconda

# Create environment
conda create -n myenv -c conda-forge -c bioconda python=3.11 <packages>

# From file
conda env create -f environment.yml

# Activate
conda activate myenv

• Use conda-forge as the primary channel, add bioconda for bioinformatics tools
• Store environments in PI storage or home — never in scratch (60-day purge)
• Use pip only as a fallback when a package is not available in conda-forge

Tools environment (recommended)

A shared tools conda env for general-purpose cluster utilities (not project-specific):

conda create -n tools -c conda-forge gh git
conda activate tools
gh auth login   # one-time setup: authenticate with GitHub

Activate tools at the start of sessions that need gh, git, or other general CLI tools. Project-specific envs (Python analysis, bioinformatics pipelines) remain separate.

R (rig + renv)

Use rig to manage R versions and renv for project-level package management, matching the local development workflow.

# Check available R versions via module system
module avail R

# Or install rig in userspace if not available as a module
# (check YCRC docs for current guidance)

# For renv projects, set cache to PI storage to avoid filling home quota:
export RENV_PATHS_CACHE="/vast/palmer/pi/musser/renv_cache"  # McCleary
# export RENV_PATHS_CACHE="/nfs/roberts/project/pi_jm284/renv_cache"  # Bouchet

# Then restore packages as usual
Rscript -e 'renv::restore()'

HPC-specific R notes:

• Some R packages need system libraries loaded via module load (e.g., HDF5, GDAL, PROJ) before renv::restore() will succeed
• Set RENV_PATHS_CACHE to PI storage so the package cache persists and doesn't eat home quota
• If using a different R version than local, renv will re-install packages for that version (the cache is version-specific)

8. Batch Processing with Snakemake

See references/snakemake.md for full Snakemake setup, SLURM executor configuration, per-rule resource specification, profile setup, and tmux usage.

When to use Snakemake: Fan-out/fan-in workflows (many samples through same steps) or complex dependency graphs. For linear pipelines (A → B → C), a simple bash script with checkpointing is fine.

Checkpointing principle: All orchestrators must check for existing output before re-running a stage. The output file itself is the checkpoint — no sentinel files.

9. Data Transfer

rsync (preferred for large transfers)

# Local to cluster
rsync -avz --progress local_dir/ <netid>@mccleary.ycrc.yale.edu:/vast/palmer/pi/musser/project/

# Cluster to local
rsync -avz --progress <netid>@mccleary.ycrc.yale.edu:/vast/palmer/pi/musser/project/results/ local_results/

scp (simple single-file transfers)

scp file.fasta <netid>@mccleary.ycrc.yale.edu:/vast/palmer/pi/musser/project/data/raw/

Globus (very large datasets)

For multi-GB transfers, use Globus (https://docs.ycrc.yale.edu/data/transfer/globus/). YCRC maintains Globus endpoints for each cluster.

Between clusters

Use Globus or direct transfer between cluster login nodes (they can reach each other).

10. YCGA Sequencing Data

Yale Center for Genome Analysis (YCGA) data is stored at /gpfs/ycga/sequencers on McCleary.

Accessing current data

YCGA sends an email with a URL when data is ready. Use the ycgaFastq utility:

module load ycga-public

# From the URL in YCGA's notification email
ycgaFastq fcb.ycga.yale.edu:3010/randomstring/sample

# By netid and flowcell
ycgaFastq <netid> AHFH66DSXX

For 10x and PacBio data, use URLFetch:

module load ycga-public
URLFetch http://fcb.ycga.yale.edu:3010/randomstring/folder

Data retention

• Retention is 6 months post-sequencing (policy effective 2026-03-31). The previous staged schedule — raw files ~45 days, fastq moved to archive ~60 days, removed from main storage ~180 days — no longer applies. Archived data persists indefinitely.

Archived data retrieval

Archives are in AWS Deep Glacier. Retrieval via web browser (http://archive.ycga.yale.edu) or ycgaFastq. Normal retrieval: 48 hours. Expedited: 12 hours (8x more expensive).

YCGA partition

Submit YCGA-related analysis jobs with -p ycga on McCleary to avoid compute charges. Eligible users: Yale PIs using YCGA for sequencing and their authorized lab members.

11. Interactive Command Conventions

When giving the user commands to run on the cluster (not batch scripts), follow these conventions:

Always background long-running commands

Append & to any command that will take more than a few seconds:

pigz -p 8 *.fastq &
fasterq-dump --split-files --threads 8 SRR123456 &

This lets the user keep working in the same shell. Mention jobs and fg for checking or re-attaching.

Prefer parallel tools

Parallel processing of many files

When operating on many independent files, use xargs -P:

ls *.fastq | xargs -P 8 -I {} gzip {} &

Always show full paths or clear context

Commands should be unambiguous — include cd to the working directory or use absolute paths so the user can copy-paste without guessing context.

12. Job Script Generation

When asked to create a SLURM job, Claude operates in one of two modes depending on where it is running.

Local mode (Claude Code on laptop)

1. Generate the batch script (.sh) in the project directory
2. Show the full script for review
3. Provide the transfer and submit commands:

   rsync -avz batch/my_job.sh <netid>@mccleary.ycrc.yale.edu:/vast/palmer/pi/musser/project/batch/
   ssh mccleary "sbatch /vast/palmer/pi/musser/project/batch/my_job.sh"
   

4. Also transfer any required input data or scripts referenced by the batch script

Cluster mode (Claude Code on compute node via interactive session)

1. Generate the batch script in the project's batch/ directory
2. Always show the full script and ask for confirmation before submitting
3. On approval, submit with sbatch batch/my_job.sh
4. Report the job ID and monitoring command (squeue --me, jobstats <id>)

The .py + .sh pattern

In data science projects, batch scripts are thin SLURM wrappers that call .py analysis scripts — the .sh holds only SLURM directives, env activation, and the python invocation; all logic lives in the .py. The wrapper is the §4 template with four deltas: set -euo pipefail, conda-only activation via source $(conda info --base)/etc/profile.d/conda.sh, a named log file (logs/slurm-<brief_name>-%j.out), and SECONDS run timing — and it drops §4's module purge + tool modules since there's no cluster-only module. See the script-organization skill for the full template and numbering convention.

Commit before submit

Always commit scripts before submitting batch jobs — the git hash in the SLURM log (and BUILD_INFO.txt) must reflect the code that actually ran; commit the .py and its .sh wrapper before sbatch. This is the cluster case of the script-organization skill's "commit before execute" convention.

Script generation rules

• Use the lab default batch script template from Section 4, including the provenance block
• Use the tool resource templates from Section 6 for SLURM directives
• Always include: --job-name, --partition, --time, --cpus-per-task, memory, --output
• Always module purge before loading modules (omit module purge when the only module is miniconda for conda-only jobs)
• Always pin module versions (e.g., module load CellRanger/9.0.1, never bare module load CellRanger)
• Follow the hybrid rule from Section 7: conda for portable tools, modules for cluster-only tools — never both for the same tool
• Always set --mail-type=BEGIN,END,FAIL and --mail-user=<your email>
• Log files go to logs/ subdirectory
• For YCGA data analysis on McCleary, use --partition=ycga
• Add comments explaining non-obvious resource choices (e.g., "128G needed for PROST structure DB")
• Log version strings for every tool actually invoked in the script (in the provenance block)
• If unsure about resources, start conservative and note "check with jobstats after first run"

13. Policies

YCRC policy hard-values live at their topical homes; this index points to them so the numbers have a single source of truth (update them there, not here):

• Scratch 60-day purge (+ no conda envs / no sole-copy raw data on scratch, email one week before, no timestamp-gaming) — see §3 Storage policies.
• Job rate limit — 200 submissions/hour — see §4 Job arrays.
• Max 4 concurrent OOD interactive apps per user — see §2 Open OnDemand.
• McCleary decommissioning 2026 / Bouchet primary going forward — see §2 Cluster Overview.
• ycga partition compute-charge exemption (-p ycga, McCleary, YCGA data) — see §2 and §10.
• Module system (module load / module avail / always module purge first) — see §7/§12.