# Slurm Auto Retry Watchdog

> Submit a Slurm sbatch job through a watchdog that monitors sacct/squeue, classifies common failures from job state and stdout/stderr logs, and automatically resubmits retriable failures up to a fixed attempt limit. Use when the user wants automatic retry for Slurm jobs, wants jobs to recover from timeout/OOM/node failures, or wants a reusable trigger/workflow around sbatch submission.

- Skill: `kwongfuk/slurm-auto-retry-watchdog` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add kwongfuk/slurm-auto-retry-watchdog`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/slurm-auto-retry-watchdog/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/slurm-auto-retry-watchdog

---


# Slurm Auto Retry Watchdog

Use this skill when a Slurm job should be submitted through a monitored retry loop instead of a one-shot `sbatch`.

## What it does

The bundled script:
- submits an `sbatch` script
- waits for terminal state via `sacct` with `squeue` fallback
- resolves stdout/stderr log paths from `#SBATCH -o/-e`
- classifies failures from job state plus log text
- retries retriable categories up to `--max-attempts`
- writes a JSON report for all attempts

## Retriable categories

Default retriable classes are:
- `timeout`
- `out_of_memory`
- `node_failure`
- `preempted`
- `launch_failure`
- `cuda_runtime`

Default non-retriable classes are:
- `python_bug`
- `import_env_bug`
- `argument_bug`
- `permission_bug`

If the user wants different behavior, pass `--retry-on` explicitly.

## Use

From the target repository root, run:

```bash
python /home/gguo/.codex/skills/slurm-auto-retry-watchdog/scripts/slurm_watchdog.py \
  slurm/run_some_job.sbatch \
  --max-attempts 5
```

Useful flags:
- `--poll-seconds 30`
- `--state-dir logs/slurm/watchdog`
- `--retry-on timeout,out_of_memory,node_failure,preempted,launch_failure,cuda_runtime`
- `--dry-run` to inspect parsed `#SBATCH` metadata without submitting

## Notes

- This workflow resubmits the same `sbatch` script. It does not patch code automatically.
- If a job repeatedly fails with a deterministic bug, the watchdog stops early and records the classified failure.
- If a script supports env overrides for batch size or model settings, add them in the `sbatch` file and then teach the watchdog to pass attempt-aware exports only when needed.