# Monitor

> Monitor Slurm jobs, Nextflow pipelines, and task arrays. Diagnose failures from logs and suggest codebase fixes.

- Skill: `theob0t/monitor` (Agent Skill)
- Install (CLI): `npx skillmds@latest add theob0t/monitor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/theob0t/monitor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: Theob0t (https://skillmd.com/u/theob0t)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/theob0t/monitor

---


# Slurm Job Monitor

You monitor Slurm HPC jobs for user `tbotella`. Gather real-time data, classify jobs, build a summary table, and deeply diagnose any failures by reading logs and tracing errors to source code.

## Step 1: Gather Live Data

### Running & Pending Jobs

!`squeue -u tbotella --format="%.12i %.50j %.8T %.12M %.12l %.6C %.10m %.12P %.40R" --sort=-M 2>&1`

### All Completed/Failed Jobs

**Important**: When a Nextflow orchestrator job is running, query sacct from the orchestrator's start time to capture the **full workflow history**, not just the last 24h. Identify the orchestrator from squeue (long-running standalone job, typically days of runtime) and derive its start time from its elapsed time.

If no orchestrator is found, or for non-Nextflow jobs, fall back to the last 24h or `$ARGUMENTS` time window.

!`sacct -u tbotella --starttime=$(date -d '72 hours ago' '+%Y-%m-%dT%H:%M:%S') --format=JobID,JobName%60,State%15,ExitCode,Elapsed,MaxRSS,Partition,AllocCPUS,ReqMem --parsable2 --noheader 2>&1`

After getting squeue output, check for a Nextflow orchestrator job (long-running, non-`nf-` job). If found, re-query sacct with `--starttime` matching the orchestrator's start time so you capture ALL jobs from the full workflow run. Example:
```bash
# If orchestrator has been running 3d 12h, query from 4 days ago to be safe
sacct -u tbotella --starttime=$(date -d '4 days ago' '+%Y-%m-%dT%H:%M:%S') --format=JobID,JobName%120,State%15,ExitCode,Elapsed,MaxRSS,Partition,AllocCPUS,ReqMem --parsable2 --noheader 2>&1
```

## Step 2: Parse & Classify Jobs

### Filtering sacct output
- **Remove sub-job entries**: Skip lines where JobID contains `.batch`, `.extern`, or `.0` — only keep main job entries.
- **Remove RUNNING entries from sacct** if they already appear in squeue (avoid double-counting).

### Job classification
1. **Nextflow jobs**: Job name starts with `nf-` (pattern: `nf-PROCESS_NAME_(sample_id)`). Group by process name.
2. **Task arrays**: JobID contains `_` separator (e.g., `12345_1`, `12345_[1-100]`). Group by base JobID.
3. **Standalone Slurm jobs**: Everything else. List individually.

### Argument handling
- No arguments or empty `$ARGUMENTS`: Default behavior — full workflow window (from orchestrator start) for NF runs, last 24h otherwise
- `$ARGUMENTS` is a numeric job ID: Focus on that specific job — show detailed info, read its logs
- `$ARGUMENTS` is "failures" or "errors": Only show FAILED/TIMEOUT/CANCELLED jobs with full diagnostics
- `$ARGUMENTS` contains a time window ("48h", "3d", "1w"): Override the sacct `--starttime` window

## Step 3: Build Summary Table

### For Nextflow runs (grouped by process):

Count **unique samples** per process (not just job attempts — a sample retried 3 times is still 1 sample). Track:
- **Samples Done**: unique samples whose latest job attempt is COMPLETED
- **Samples Total**: unique samples that have been submitted to this process at least once
- **Running/Pending**: current job counts from squeue
- **Failed (jobs)**: total failed job attempts (includes retries)
- **Progress %**: Samples Done / Samples Total

Also report **Total Unique Samples** across the entire workflow (union of all sample IDs seen in any process).

| # | Process | Samples Done | Samples Total | Running | Failed (jobs) | Progress | Notes |
|---|---------|-------------|---------------|---------|---------------|----------|-------|

### For task arrays (grouped by base JobID):

| # | Job Name | ArrayID | Running | Pending | Done | Failed | Runtime | Notes |
|---|----------|---------|---------|---------|------|--------|---------|-------|

### For standalone jobs:

| # | Job Name | JobID | Status | Runtime | Exit | Notes |
|---|----------|-------|--------|---------|------|-------|

### Summary line:
```
Running: X | Pending: Y | Completed: Z | Failed: W | Cancelled: V
```

## Step 4: Nextflow Pipeline Progress

If Nextflow jobs are detected:

### 4a. Identify the orchestrator and active log
Find the Nextflow orchestrator job in squeue (long-running non-`nf-` job). Then locate the active `.nextflow.log` — check `launch_*/` subdirectories under the pipeline root, sorted by modification time:
```bash
ls -lt STREGA/launch_*/.nextflow.log 2>/dev/null | head -5
```
Read the run name and session from the most recently modified log:
```bash
grep -E "Run name:|Session" <log_path> | tail -4
```

### 4b. Compute unique sample progress from sacct
Parse the full sacct output to extract process name and sample ID from each `nf-PROCESS_NAME_(sample_id)` job. For each process, count:
- Unique samples submitted (total distinct sample IDs)
- Unique samples completed (latest attempt for that sample is COMPLETED)
- Failed job attempts (total, including retries of the same sample)
- Report the total unique sample count across all processes

### 4c. Check execution trace (optional)
If a recent trace file exists, cross-reference with sacct data:
```bash
find $(pwd)/results -name "execution_trace*" -mmin -1440 -type f 2>/dev/null | head -5
```

## Step 5: Deep Failure Diagnosis

**This is the most important step.** For EVERY failed job, do all of the following:

### 5a. Find and read the error logs

For **Nextflow failures**:
1. Search `.nextflow.log` for the failed task's work directory hash:
   ```bash
   grep -E "FAILED|Error executing process" .nextflow.log | tail -20
   ```
2. Once you have the work directory path (e.g., `work/ab/cd1234.../`), read the logs:
   ```bash
   cat work/ab/cd1234*/.exitcode
   tail -80 work/ab/cd1234*/.command.log
   tail -40 work/ab/cd1234*/.command.err
   ```
3. Also read `.command.sh` to understand what command was run.

For **standalone Slurm jobs**:
1. Find the log file from the sbatch `--output` pattern (typically `logs/%x_%A_%a.log`).
2. Read the last 80 lines of the log.

For **task arrays**: Read logs for the specific failed array indices.

### 5b. Parse the actual error message

Look for these patterns in the logs:
- Python tracebacks: Find the last `Traceback (most recent call last)` block and the final exception line
- Java exceptions: `Exception in thread`, `java.lang.OutOfMemoryError`, GATK errors
- STAR errors: `EXITING because of FATAL ERROR in input reads`
- Filesystem errors: `No space left on device`, `FileNotFoundException`, `No such file`
- Memory errors: `Cannot allocate memory`, `MemoryError`, `Killed`
- Container errors: `FATAL`, `singularity/apptainer` error lines

Extract the **specific error message** — not just "it failed".

### 5c. Trace error to source code

Use Grep and Read to find the relevant source code that produced the error:

- **Nextflow modules**: `modules/` directory — find the process definition to understand what command was run and what resources were allocated
- **Python scripts**: `STREGA/` directory — if it's a Python traceback, find the exact file and line mentioned in the traceback. Key scripts:
  - `STREGA/readLevel.py` — read-level feature extraction
  - `STREGA/posLevel.py` — position-level features
  - `STREGA/combineData.py` — data merging
  - `STREGA/RunLearning.py` — ML classification
  - `STREGA/enrichmentPreprocessing.py` — enrichment pipeline
  - `STREGA/enrichment_utils.py` — enrichment utilities
- **Config files**: `conf/base.config` for resource allocations, `nextflow.config` for params

### 5d. Suggest a concrete fix

Based on the error + source code context, provide a **specific, actionable fix**:

| Error Type | Diagnosis Approach | Fix Format |
|-----------|-------------------|------------|
| OOM (exit 137) | Check MaxRSS in sacct, find the process in `conf/base.config` | "Increase memory in `conf/base.config` line N: `X.GB` → `Y.GB`" |
| Timeout (exit 143) | Check Elapsed vs TimeLimit, find process time config | "Increase time in `conf/base.config` line N: `'Xh'` → `'Yh'`" |
| Python traceback | Read the traceback, find the file:line, understand the bug | "Fix in `STREGA/script.py` line N: [specific code change]" |
| Missing input file | Check if upstream task failed, check path in config | "Upstream task PROCESS_NAME failed — fix that first" or "Path X does not exist" |
| Disk full | Check scratch usage | "Clean work directory: `nextflow clean -f -before <run_name>`" |
| Container error | Check if tool is available in the SIF | "Tool X missing from container Y — rebuild or use different container" |

### 5e. Prepare rerun command

For Nextflow:
```bash
conda run -n strega-hpc --no-capture-output nextflow run main.nf -profile hpc -c conf/local_sif.config -params-file <params.yaml> -resume
```

For standalone Slurm: Show the modified `sbatch` command with fixed resources.

For task arrays: Show how to resubmit only the failed indices.

## Step 6: Output Format

Present results in this order:

1. **Pipeline overview** (if Nextflow): Run name, session, overall progress
2. **Status table(s)**: Grouped by type (NF processes, arrays, standalone)
3. **Summary counts line**
4. **Failure diagnostics** (if any): One section per failed job/process with:
   ```
   ### FAILED: PROCESS_NAME (sample_id)
   - Work dir: work/ab/cd1234...
   - Exit code: 137 (OOM — killed by scheduler, MaxRSS: 23.5G vs 24G limit)
   - Error: "java.lang.OutOfMemoryError: Java heap space"
   - Source: modules/preprocessing/split_cigar.nf → GATK SplitNCigarReads
   - Fix: Increase memory in conf/base.config line 45: 24.GB → 48.GB
   - Rerun: nextflow run main.nf ... -resume
   ```

If no jobs are found:
> No running or pending jobs. No completed/failed jobs found.

## Exit Code Quick Reference

| Exit | Signal | Cause |
|------|--------|-------|
| 137 | SIGKILL | OOM — process exceeded memory limit |
| 143 | SIGTERM | Timeout — exceeded wall time |
| 140 | SIGKILL | Killed by scheduler |
| 134 | SIGABRT | Application abort/assertion failure |
| 104 | — | Application fatal error (common in STAR) |
| 1 | — | General application failure |
| 0:9 | SIGKILL | OOM (sacct ExitCode:Signal format) |