Slurm Auto Retry Watchdog

Slurm Auto Retry Watchdog

Use this skill when a Slurm job should be submitted through a monitored retry loop instead of a one-shot sbatch.

What it does

The bundled script:

• submits an sbatch script
• waits for terminal state via sacct with squeue fallback
• resolves stdout/stderr log paths from #SBATCH -o/-e
• classifies failures from job state plus log text
• retries retriable categories up to --max-attempts
• writes a JSON report for all attempts

Retriable categories

Default retriable classes are:

• timeout
• out_of_memory
• node_failure
• preempted
• launch_failure
• cuda_runtime

Default non-retriable classes are:

• python_bug
• import_env_bug
• argument_bug
• permission_bug

If the user wants different behavior, pass --retry-on explicitly.

Use

From the target repository root, run:

python /home/gguo/.codex/skills/slurm-auto-retry-watchdog/scripts/slurm_watchdog.py \
  slurm/run_some_job.sbatch \
  --max-attempts 5

Useful flags:

• --poll-seconds 30
• --state-dir logs/slurm/watchdog
• --retry-on timeout,out_of_memory,node_failure,preempted,launch_failure,cuda_runtime
• --dry-run to inspect parsed #SBATCH metadata without submitting

Notes

• This workflow resubmits the same sbatch script. It does not patch code automatically.
• If a job repeatedly fails with a deterministic bug, the watchdog stops early and records the classified failure.
• If a script supports env overrides for batch size or model settings, add them in the sbatch file and then teach the watchdog to pass attempt-aware exports only when needed.