Molgpu

MolGPU Cluster Reference

Shared GPU cluster for the group. All nodes on private network (use MIT VPN to connect off campus).

Machines

Quick Selection Guide

• 4x GPU (fastest): molgpu07 (4x 4090, best single-GPU perf + most cores)
• 4x GPU (fallback): molgpu02 or molgpu03 (4x A5000)
• 2x GPU: molgpu05 or molgpu06 (2x 4090)
• Many small jobs: molgpu08 (8x 2080 Ti, only 11 GB each)

Access & SSH Setup

Full hostnames are molgpu01.mit.edu through molgpu08.mit.edu. Same credentials across all nodes. SSH keys set up on one node work on all others (shared home dir).

Add this to ~/.ssh/config for easy access:

Host *
  AddKeysToAgent yes
  UseKeychain yes   # macOS only — saves key password in keychain
  ForwardAgent yes

Host molgpu0?
  Hostname %h.mit.edu
  User <your-username>
  ForwardAgent yes
  IdentityFile <path/to/private_key>

This lets you ssh molgpu07 instead of ssh user@molgpu07.mit.edu -i /path/to/key. ForwardAgent means you don't need to store your private key on the server to use GitHub via SSH.

To add access from a new client, append your public key to ~/.ssh/authorized_keys on any molgpu node.

Change password with passwd (applies to all nodes, must be mixed case + numeric, 8+ chars).

Shared Filesystem

All molgpu machines mount the same NFS home directory at /mnt/home/ (81.8 TB total). Files written on one machine are immediately visible on all others — no rsync needed between molgpus. Only rsync from your local machine to any one node.

Sensitive data goes on the encrypted partition at /mnt/encrypted/ (16.4 TB). Protect sensitive folders with chmod 700.

Important: Avoid reading/writing a large number of files in parallel — this slows NFS for everyone. Reduce parallel I/O workers if NFS feels slow.

Always back up code and important checkpoints. There is no guarantee storage will never fail.

Conda / Mamba

Global conda is available on molgpu02–06 (no need to install your own):

/opt/miniconda3/bin/conda init

Global mamba (faster drop-in replacement for conda) on molgpu02:

# after conda init:
mamba init

uv is also available at ~/.local/bin/uv if installed per-user.

Dashboard

View current load across all machines: http://molgpu01:8088/

Common Commands

# transfer files from local machine (only need to hit one node, shared fs does the rest)
rsync -az --exclude '.venv*' --exclude '__pycache__' --exclude '.git' \
  . molgpu07:~/project-name/

# check GPU availability on a node
ssh molgpu07 nvidia-smi --query-gpu=index,name,memory.used,memory.free --format=csv,noheader

# scan all machines at once
for h in molgpu0{1..8}; do echo "=== $h ===" && ssh $h nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader 2>/dev/null; done

# always use tmux for long-running jobs (nohup is NOT enough — child processes get SIGHUP)
ssh molgpu07
tmux new-session -s myjob
# ... launch job ...
# detach: Ctrl-B then D
# reattach later: tmux attach -t myjob

# ONE-TIME per user per node: enable linger so the tmux server survives
# when your login session ends. without this, systemd-logind kills the
# whole user slice (including detached tmux sessions) when your last
# ssh closes.
loginctl enable-linger
loginctl show-user $USER --property=Linger  # should read Linger=yes

# always redirect stderr (some frameworks log errors only to stderr)
python train.py > /tmp/train.log 2>&1

# kill orphaned GPU processes after a crash
nvidia-smi  # find PIDs
kill -9 <pid>

Known Issues

• NFS stale file handles. Caches stored in home dir (~/.triton/cache, ~/.cache/torch/) can go stale when switching between machines. Fix: use local disk for caches (export TRITON_CACHE_DIR=/tmp/triton_cache) or nuke the cache (rm -rf ~/.triton/cache).
• All GPUs are 24 GB (except molgpu08 at 11 GB).
• Torch / CUDA version mismatch. NVIDIA driver on molgpu nodes is CUDA 12.4 (as of 2026-04). Recent PyTorch (2.10+) ships only cu130 wheels and will silently fall back to CPU with a driver-too-old warning. If torch.cuda.is_available() is False after a fresh install, pin torch to a cu124 build:

  ~/.local/bin/uv pip install --force-reinstall torch==2.5.1 \
      --index-url https://download.pytorch.org/whl/cu124
  

  Verify with python -c "import torch; print(torch.cuda.is_available())".

LLM/VLM-Specific Notes

Multi-GPU Training (DDP)

# set HF_HUB_OFFLINE=1 to avoid HF Hub race conditions during multi-GPU load
# (download model first, then set this)
HF_HUB_OFFLINE=1 accelerate launch --num_processes=4 train.py

# QLoRA + DDP: use device_map={"": local_rank} (not None or "auto")

vLLM Inference on 24 GB GPUs

VLMs are especially memory-hungry. If OOM during warmup or generation, tune:

• gpu_memory_utilization: lower from 0.9 to 0.35–0.7
• max_num_seqs: lower from 256 to 4–16
• enforce_eager=True: skips CUDA graphs, saves ~2 GB VRAM
• tensor_parallel_size: split model across GPUs

Model Size Reference