Monitor

Slurm Job Monitor

You monitor Slurm HPC jobs for user tbotella. Gather real-time data, classify jobs, build a summary table, and deeply diagnose any failures by reading logs and tracing errors to source code.

Step 1: Gather Live Data

Running & Pending Jobs

!squeue -u tbotella --format="%.12i %.50j %.8T %.12M %.12l %.6C %.10m %.12P %.40R" --sort=-M 2>&1

All Completed/Failed Jobs

Important: When a Nextflow orchestrator job is running, query sacct from the orchestrator's start time to capture the full workflow history, not just the last 24h. Identify the orchestrator from squeue (long-running standalone job, typically days of runtime) and derive its start time from its elapsed time.

If no orchestrator is found, or for non-Nextflow jobs, fall back to the last 24h or $ARGUMENTS time window.

!sacct -u tbotella --starttime=$(date -d '72 hours ago' '+%Y-%m-%dT%H:%M:%S') --format=JobID,JobName%60,State%15,ExitCode,Elapsed,MaxRSS,Partition,AllocCPUS,ReqMem --parsable2 --noheader 2>&1

After getting squeue output, check for a Nextflow orchestrator job (long-running, non-nf- job). If found, re-query sacct with --starttime matching the orchestrator's start time so you capture ALL jobs from the full workflow run. Example:

# If orchestrator has been running 3d 12h, query from 4 days ago to be safe
sacct -u tbotella --starttime=$(date -d '4 days ago' '+%Y-%m-%dT%H:%M:%S') --format=JobID,JobName%120,State%15,ExitCode,Elapsed,MaxRSS,Partition,AllocCPUS,ReqMem --parsable2 --noheader 2>&1

Step 2: Parse & Classify Jobs

Filtering sacct output

• Remove sub-job entries: Skip lines where JobID contains .batch, .extern, or .0 — only keep main job entries.
• Remove RUNNING entries from sacct if they already appear in squeue (avoid double-counting).

Job classification

1. Nextflow jobs: Job name starts with nf- (pattern: nf-PROCESS_NAME_(sample_id)). Group by process name.
2. Task arrays: JobID contains _ separator (e.g., 12345_1, 12345_[1-100]). Group by base JobID.
3. Standalone Slurm jobs: Everything else. List individually.

Argument handling

• No arguments or empty $ARGUMENTS: Default behavior — full workflow window (from orchestrator start) for NF runs, last 24h otherwise
• $ARGUMENTS is a numeric job ID: Focus on that specific job — show detailed info, read its logs
• $ARGUMENTS is "failures" or "errors": Only show FAILED/TIMEOUT/CANCELLED jobs with full diagnostics
• $ARGUMENTS contains a time window ("48h", "3d", "1w"): Override the sacct --starttime window

Step 3: Build Summary Table

For Nextflow runs (grouped by process):

Count unique samples per process (not just job attempts — a sample retried 3 times is still 1 sample). Track:

• Samples Done: unique samples whose latest job attempt is COMPLETED
• Samples Total: unique samples that have been submitted to this process at least once
• Running/Pending: current job counts from squeue
• Failed (jobs): total failed job attempts (includes retries)
• Progress %: Samples Done / Samples Total

Also report Total Unique Samples across the entire workflow (union of all sample IDs seen in any process).

For task arrays (grouped by base JobID):

For standalone jobs:

Summary line:

Running: X | Pending: Y | Completed: Z | Failed: W | Cancelled: V

Step 4: Nextflow Pipeline Progress

If Nextflow jobs are detected:

4a. Identify the orchestrator and active log

Find the Nextflow orchestrator job in squeue (long-running non-nf- job). Then locate the active .nextflow.log — check launch_*/ subdirectories under the pipeline root, sorted by modification time:

ls -lt STREGA/launch_*/.nextflow.log 2>/dev/null | head -5

Read the run name and session from the most recently modified log:

grep -E "Run name:|Session" <log_path> | tail -4

4b. Compute unique sample progress from sacct

Parse the full sacct output to extract process name and sample ID from each nf-PROCESS_NAME_(sample_id) job. For each process, count:

• Unique samples submitted (total distinct sample IDs)
• Unique samples completed (latest attempt for that sample is COMPLETED)
• Failed job attempts (total, including retries of the same sample)
• Report the total unique sample count across all processes

4c. Check execution trace (optional)

If a recent trace file exists, cross-reference with sacct data:

find $(pwd)/results -name "execution_trace*" -mmin -1440 -type f 2>/dev/null | head -5

Step 5: Deep Failure Diagnosis

This is the most important step. For EVERY failed job, do all of the following:

5a. Find and read the error logs

For Nextflow failures:

1. Search .nextflow.log for the failed task's work directory hash:

   grep -E "FAILED|Error executing process" .nextflow.log | tail -20
   

2. Once you have the work directory path (e.g., work/ab/cd1234.../), read the logs:

   cat work/ab/cd1234*/.exitcode
   tail -80 work/ab/cd1234*/.command.log
   tail -40 work/ab/cd1234*/.command.err
   

3. Also read .command.sh to understand what command was run.

For standalone Slurm jobs:

1. Find the log file from the sbatch --output pattern (typically logs/%x_%A_%a.log).
2. Read the last 80 lines of the log.

For task arrays: Read logs for the specific failed array indices.

5b. Parse the actual error message

Look for these patterns in the logs:

• Python tracebacks: Find the last Traceback (most recent call last) block and the final exception line
• Java exceptions: Exception in thread, java.lang.OutOfMemoryError, GATK errors
• STAR errors: EXITING because of FATAL ERROR in input reads
• Filesystem errors: No space left on device, FileNotFoundException, No such file
• Memory errors: Cannot allocate memory, MemoryError, Killed
• Container errors: FATAL, singularity/apptainer error lines

Extract the specific error message — not just "it failed".

5c. Trace error to source code

Use Grep and Read to find the relevant source code that produced the error:

• Nextflow modules: modules/ directory — find the process definition to understand what command was run and what resources were allocated
• Python scripts: STREGA/ directory — if it's a Python traceback, find the exact file and line mentioned in the traceback. Key scripts:
  • STREGA/readLevel.py — read-level feature extraction
  • STREGA/posLevel.py — position-level features
  • STREGA/combineData.py — data merging
  • STREGA/RunLearning.py — ML classification
  • STREGA/enrichmentPreprocessing.py — enrichment pipeline
  • STREGA/enrichment_utils.py — enrichment utilities
• Config files: conf/base.config for resource allocations, nextflow.config for params

5d. Suggest a concrete fix

Based on the error + source code context, provide a specific, actionable fix:

5e. Prepare rerun command

For Nextflow:

conda run -n strega-hpc --no-capture-output nextflow run main.nf -profile hpc -c conf/local_sif.config -params-file <params.yaml> -resume

For standalone Slurm: Show the modified sbatch command with fixed resources.

For task arrays: Show how to resubmit only the failed indices.

Step 6: Output Format

Present results in this order:

1. Pipeline overview (if Nextflow): Run name, session, overall progress
2. Status table(s): Grouped by type (NF processes, arrays, standalone)
3. Summary counts line
4. Failure diagnostics (if any): One section per failed job/process with:

   ### FAILED: PROCESS_NAME (sample_id)
   - Work dir: work/ab/cd1234...
   - Exit code: 137 (OOM — killed by scheduler, MaxRSS: 23.5G vs 24G limit)
   - Error: "java.lang.OutOfMemoryError: Java heap space"
   - Source: modules/preprocessing/split_cigar.nf → GATK SplitNCigarReads
   - Fix: Increase memory in conf/base.config line 45: 24.GB → 48.GB
   - Rerun: nextflow run main.nf ... -resume
   

If no jobs are found:

No running or pending jobs. No completed/failed jobs found.

Exit Code Quick Reference