Monitor HPC Jobs
You are monitoring HPC jobs on a cluster. Follow this workflow:
Step 0: Resolve project root and environment
- Look for
.env by walking up from $PWD (check $PWD/.env, $PWD/../.env, etc.)
- The directory containing
.env is $PROJECT_ROOT
- If no
.env is found, fall back to $PWD
- Read
.env to get JOBS_DIR (used for locating job scripts and logs).
Step 1: Detect the scheduler and cluster
command -v sbatch && echo "SLURM" || (command -v qsub && echo "PBS" || echo "UNKNOWN")
Then identify which cluster you are on:
# CX3 (Imperial) — PBS Pro
[[ "$PWD" == /rds/general/user/* ]] && echo "CX3"
# Isambard — SLURM
[[ "$PWD" == /home/*/* ]] && ls /projects/ &>/dev/null && echo "ISAMBARD"
If on CX3, also check if you are on a login node (hostname contains login). If you are NOT on a login node, you can run short Python commands directly without submitting a job.
Follow the SLURM or PBS Pro paths below depending on the scheduler.
Step 2: Identify the job
- If
$ARGUMENTS contains a job ID, use that.
SLURM
- Run
squeue -u $USER --format="%.18i %.20j %.8T %.10M %.6D %.4C %.20R %o" --sort=-V to find the user's active jobs.
- If no jobs are running/pending, check recently completed jobs with
sacct -u $USER --starttime=$(date -d '24 hours ago' +%Y-%m-%d) --format=JobID,JobName%30,State,ExitCode,Start,End,Elapsed --noheader.
PBS Pro (CX3)
- Run
qstat -u $USER to find the user's active jobs.
- If no jobs are running/queued, try
qstat -f <job_id> if a job ID was provided (may still work for recently-finished jobs). Otherwise, report that no active jobs were found — PBS Pro has no sacct-equivalent history command.
Step 3: Get job details
SLURM
- Run
scontrol show job <job_id> to get full job info (WorkDir, StdOut path, submission script, state, node allocation).
- Extract the log file path from the
StdOut field. Note: %j in the path is replaced by the job ID.
PBS Pro (CX3)
- Run
qstat -f <job_id> to get full job info (Output_Path, Error_Path, job state, resources).
- Extract both log paths:
Output_Path (stdout) and Error_Path (stderr). PBS Pro uses separate files unlike SLURM.
- PBS log filenames are fixed strings — there is no
%j job ID expansion.
- Log paths from
#PBS -o / #PBS -e are relative to PBS_O_WORKDIR (the submission directory). Resolve them accordingly.
Step 4: Wait for job to start (if PENDING/QUEUED)
If the job is PENDING (SLURM) or Q (PBS), do NOT just report and stop. Instead, automatically poll until it starts:
- Report the current state: "Job 12345 is PENDING. Waiting for it to start..."
- Run
sleep 30 in the background using run_in_background: true.
- After each sleep, re-check:
- SLURM:
squeue -j <job_id> --noheader --format="%.18i %.8T"
- PBS:
qstat <job_id> (check the state column)
- If still pending, report briefly ("Still pending, waited Xm so far...") and sleep again.
- Repeat until the job transitions to RUNNING (R) or FAILED/CANCELLED.
- Once RUNNING, wait 10 more seconds for the log file to appear, then proceed to Step 5.
This polling loop is the default behavior — do not ask the user whether to wait. Just do it.
Step 5: Monitor the logs
- Check if the log file exists yet. If not, sleep 10 seconds and retry (up to 5 times).
- Once the log file exists, read the last 100 lines using
tail -n 100 <log_path>.
- PBS Pro: Check both the
.out (stdout) and .err (stderr) log files. Errors often appear only in the .err file.
- Look for:
- Errors: Python tracebacks,
Error, Exception, FAILED, CANCELLED, OOM, CUDA error, RuntimeError
- Warnings:
UserWarning, FutureWarning, deprecation notices
- Progress indicators: epoch numbers, step counts, loss values, W&B run URLs
- Training metrics:
train/loss, val/loss, learning rate, gradient norms
Step 6: Report status
Provide a concise summary:
- Job state (PENDING / RUNNING / COMPLETED / FAILED / TIMEOUT)
- Runtime so far
- Training progress (current epoch/step, latest loss values)
- W&B run URL if visible in logs
- Errors or warnings if any — include the full traceback
Step 7: If errors are found
- Analyze the error and identify the root cause.
- Check the relevant source files in the codebase to understand the issue.
- Propose a fix. If the fix is clear and safe:
- Apply the fix to the source code.
- Cancel the failed job (ask user first):
- SLURM:
scancel <job_id>
- PBS:
qdel <job_id>
- Resubmit the job with the original script.
- If the fix is ambiguous or risky, present options to the user before acting.
Step 8: Continuous monitoring (if job is still running)
If the job is running and healthy, offer to keep monitoring. If the user wants continued monitoring:
- Sleep for 60 seconds between checks.
- Re-read the last 100 lines of the log.
- Report any new errors, warnings, or progress updates.
- Stop monitoring when the job completes, fails, or the user asks to stop.
Key paths
| What |
Path |
| Job scripts |
$JOBS_DIR/<project>/[<group>/] (the optional <group>/ sub-dir is used by /softnano:submit-wandb-job) |
| SLURM logs |
$JOBS_DIR/<project>/[<group>/]logs/ (or as specified by #SBATCH --output) |
| PBS logs |
$JOBS_DIR/<project>/[<group>/]logs/ (or as specified by #PBS -o / #PBS -e) |
| Env vars |
$PROJECT_ROOT/.env → JOBS_DIR |
CX3 (PBS Pro) Reference
Queue Reference
| Queue |
Walltime |
CPUs |
Memory |
GPUs |
Use case |
v1_gpu72 |
72h |
1–64 |
up to 920 GB |
L40S / A100 |
GPU inference/training |
v1_medium24 |
24h |
1–64 |
up to 450 GB |
— |
CPU-only, moderate memory |
v1_largemem72 |
72h |
1–128 |
921–4000 GB |
— |
mmseqs2 / large-index jobs |
12 GPU limit per user on v1_gpu72.
CX3 Storage
| Path |
Size |
Notes |
/rds/general/user/<user>/home |
1 TB |
Code, logs, outputs |
/rds/general/user/<user>/ephemeral |
10 TB |
Large datasets (auto-deleted every 30 days) |
Key Differences from SLURM
- No
srun — run Python directly after activating the venv.
- No
%j job ID expansion in #PBS -o — use fixed log filenames.
- Resource syntax:
select=<N>:ncpus=<C>:mem=<M>gb:ngpus=<G> instead of --nodes/--gpus.
- Variables via
qsub -v "KEY=val" instead of --export.
- Queue via
#PBS -q instead of #SBATCH --partition.
Important notes
- SLURM: Always resolve
%j in log paths to the actual job ID.
- PBS Pro: Monitor both
.out and .err files. Log filenames are fixed (no job ID expansion).
- If the log path is relative, resolve it relative to the job's WorkDir (SLURM: from
scontrol, PBS: from PBS_O_WORKDIR).
- When sleeping/waiting, use
sleep in Bash with run_in_background to avoid blocking.
- Present errors with enough context (surrounding lines) for the user to understand.
- See the
cluster-instructions skill (isambard.md for SLURM, cx3.md for PBS Pro) for full HPC reference.
1---2name: monitor-jobs3description: Monitor SLURM/PBS jobs and their logs. Use when the user asks to monitor, check, or watch a submitted job. Automatically detects running/pending jobs, tails logs, and reports errors.4argument-hint: "[job-id (optional)]"5allowed-tools: Bash, Read, Grep, Glob, Task6metadata:7 source: https://skillmd.com/skills/softnanolab/monitor-jobs8---910# Monitor HPC Jobs11You are monitoring HPC jobs on a cluster. Follow this workflow:1213## Step 0: Resolve project root and environment14151. Look for `.env` by walking up from `$PWD` (check `$PWD/.env`, `$PWD/../.env`, etc.)162. The directory containing `.env` is `$PROJECT_ROOT`173. If no `.env` is found, fall back to `$PWD`184. Read `.env` to get `JOBS_DIR` (used for locating job scripts and logs).1920## Step 1: Detect the scheduler and cluster2122```bash23command -v sbatch && echo "SLURM" || (command -v qsub && echo "PBS" || echo "UNKNOWN")24```2526Then identify which cluster you are on:2728```bash29# CX3 (Imperial) — PBS Pro30[[ "$PWD" == /rds/general/user/* ]] && echo "CX3"3132# Isambard — SLURM33[[ "$PWD" == /home/*/* ]] && ls /projects/ &>/dev/null && echo "ISAMBARD"34```3536If on **CX3**, also check if you are on a login node (hostname contains `login`). If you are NOT on a login node, you can run short Python commands directly without submitting a job.3738Follow the **SLURM** or **PBS Pro** paths below depending on the scheduler.3940## Step 2: Identify the job4142- If `$ARGUMENTS` contains a job ID, use that.4344### SLURM45- Run `squeue -u $USER --format="%.18i %.20j %.8T %.10M %.6D %.4C %.20R %o" --sort=-V` to find the user's active jobs.46- If no jobs are running/pending, check recently completed jobs with `sacct -u $USER --starttime=$(date -d '24 hours ago' +%Y-%m-%d) --format=JobID,JobName%30,State,ExitCode,Start,End,Elapsed --noheader`.4748### PBS Pro (CX3)49- Run `qstat -u $USER` to find the user's active jobs.50- If no jobs are running/queued, try `qstat -f <job_id>` if a job ID was provided (may still work for recently-finished jobs). Otherwise, report that no active jobs were found — PBS Pro has no `sacct`-equivalent history command.5152## Step 3: Get job details5354### SLURM55- Run `scontrol show job <job_id>` to get full job info (WorkDir, StdOut path, submission script, state, node allocation).56- Extract the **log file path** from the `StdOut` field. Note: `%j` in the path is replaced by the job ID.5758### PBS Pro (CX3)59- Run `qstat -f <job_id>` to get full job info (Output_Path, Error_Path, job state, resources).60- Extract **both** log paths: `Output_Path` (stdout) and `Error_Path` (stderr). PBS Pro uses separate files unlike SLURM.61- PBS log filenames are fixed strings — there is no `%j` job ID expansion.62- Log paths from `#PBS -o` / `#PBS -e` are relative to `PBS_O_WORKDIR` (the submission directory). Resolve them accordingly.6364## Step 4: Wait for job to start (if PENDING/QUEUED)6566If the job is PENDING (SLURM) or Q (PBS), do NOT just report and stop. Instead, **automatically poll until it starts**:67681. Report the current state: "Job 12345 is PENDING. Waiting for it to start..."692. Run `sleep 30` in the background using `run_in_background: true`.703. After each sleep, re-check:71 - **SLURM**: `squeue -j <job_id> --noheader --format="%.18i %.8T"`72 - **PBS**: `qstat <job_id>` (check the state column)734. If still pending, report briefly ("Still pending, waited Xm so far...") and sleep again.745. Repeat until the job transitions to RUNNING (R) or FAILED/CANCELLED.756. Once RUNNING, wait 10 more seconds for the log file to appear, then proceed to Step 5.7677This polling loop is the **default behavior** — do not ask the user whether to wait. Just do it.7879## Step 5: Monitor the logs8081- Check if the log file exists yet. If not, sleep 10 seconds and retry (up to 5 times).82- Once the log file exists, read the **last 100 lines** using `tail -n 100 <log_path>`.83- **PBS Pro**: Check both the `.out` (stdout) and `.err` (stderr) log files. Errors often appear only in the `.err` file.84- Look for:85 - **Errors**: Python tracebacks, `Error`, `Exception`, `FAILED`, `CANCELLED`, `OOM`, `CUDA error`, `RuntimeError`86 - **Warnings**: `UserWarning`, `FutureWarning`, deprecation notices87 - **Progress indicators**: epoch numbers, step counts, loss values, W&B run URLs88 - **Training metrics**: `train/loss`, `val/loss`, learning rate, gradient norms8990## Step 6: Report status9192Provide a concise summary:931. **Job state** (PENDING / RUNNING / COMPLETED / FAILED / TIMEOUT)942. **Runtime** so far953. **Training progress** (current epoch/step, latest loss values)964. **W&B run URL** if visible in logs975. **Errors or warnings** if any — include the full traceback9899## Step 7: If errors are found100101- Analyze the error and identify the root cause.102- Check the relevant source files in the codebase to understand the issue.103- Propose a fix. If the fix is clear and safe:104 1. Apply the fix to the source code.105 2. Cancel the failed job (ask user first):106 - **SLURM**: `scancel <job_id>`107 - **PBS**: `qdel <job_id>`108 3. Resubmit the job with the original script.109- If the fix is ambiguous or risky, present options to the user before acting.110111## Step 8: Continuous monitoring (if job is still running)112113If the job is running and healthy, offer to keep monitoring. If the user wants continued monitoring:114- Sleep for 60 seconds between checks.115- Re-read the last 100 lines of the log.116- Report any new errors, warnings, or progress updates.117- Stop monitoring when the job completes, fails, or the user asks to stop.118119## Key paths120121| What | Path |122|------|------|123| Job scripts | `$JOBS_DIR/<project>/[<group>/]` (the optional `<group>/` sub-dir is used by `/softnano:submit-wandb-job`) |124| SLURM logs | `$JOBS_DIR/<project>/[<group>/]logs/` (or as specified by `#SBATCH --output`) |125| PBS logs | `$JOBS_DIR/<project>/[<group>/]logs/` (or as specified by `#PBS -o` / `#PBS -e`) |126| Env vars | `$PROJECT_ROOT/.env` → `JOBS_DIR` |127128## CX3 (PBS Pro) Reference129130### Queue Reference131132| Queue | Walltime | CPUs | Memory | GPUs | Use case |133|-------|----------|------|--------|------|----------|134| `v1_gpu72` | 72h | 1–64 | up to 920 GB | L40S / A100 | GPU inference/training |135| `v1_medium24` | 24h | 1–64 | up to 450 GB | — | CPU-only, moderate memory |136| `v1_largemem72` | 72h | 1–128 | 921–4000 GB | — | mmseqs2 / large-index jobs |137138> 12 GPU limit per user on `v1_gpu72`.139140### CX3 Storage141142| Path | Size | Notes |143|------|------|-------|144| `/rds/general/user/<user>/home` | 1 TB | Code, logs, outputs |145| `/rds/general/user/<user>/ephemeral` | 10 TB | Large datasets (auto-deleted every 30 days) |146147### Key Differences from SLURM148149- No `srun` — run Python directly after activating the venv.150- No `%j` job ID expansion in `#PBS -o` — use fixed log filenames.151- Resource syntax: `select=<N>:ncpus=<C>:mem=<M>gb:ngpus=<G>` instead of `--nodes`/`--gpus`.152- Variables via `qsub -v "KEY=val"` instead of `--export`.153- Queue via `#PBS -q` instead of `#SBATCH --partition`.154155## Important notes156157- **SLURM**: Always resolve `%j` in log paths to the actual job ID.158- **PBS Pro**: Monitor both `.out` and `.err` files. Log filenames are fixed (no job ID expansion).159- If the log path is relative, resolve it relative to the job's WorkDir (SLURM: from `scontrol`, PBS: from `PBS_O_WORKDIR`).160- When sleeping/waiting, use `sleep` in Bash with `run_in_background` to avoid blocking.161- Present errors with enough context (surrounding lines) for the user to understand.162- See the `cluster-instructions` skill (`isambard.md` for SLURM, `cx3.md` for PBS Pro) for full HPC reference.