# Slurm Self Healing Watchdog

> Submit a Slurm sbatch job through a watchdog that waits for terminal state, diagnoses failures from sacct/squeue plus stdout/stderr, invokes Codex to apply a minimal repair when needed, and resubmits up to 5 times. Use when the user wants a trigger-like loop that can monitor stopped jobs, analyze why they failed, patch the repo or sbatch script, and retry automatically.

- Skill: `kwongfuk/slurm-self-healing-watchdog` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add kwongfuk/slurm-self-healing-watchdog`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/slurm-self-healing-watchdog/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/slurm-self-healing-watchdog

---


# Slurm Self Healing Watchdog

Use this skill when a Slurm job should not stop after one failed `sbatch`.

## What It Does

The bundled script:
- submits an `sbatch` script
- polls `sacct` and falls back to `squeue`
- resolves stdout/stderr paths from `#SBATCH -o/-e`
- classifies the failure mode from Slurm state plus log text
- calls `codex exec` with a focused repair prompt when the job did not complete
- resubmits the job after repair, up to `--max-attempts 5`
- writes a JSON report for all attempts and repair summaries

## When To Use

Use this skill when the user wants:
- a post-submit trigger that watches whether a Slurm job stopped
- automatic retry after timeout, OOM, node failure, launcher failure, or deterministic script bugs
- Codex to inspect logs and make a minimal fix before resubmission
- a reusable wrapper around `sbatch` rather than a one-off manual loop

Do not use it when:
- the user only wants a plain one-shot submit
- the failure requires human judgment beyond safe local edits
- the repository must remain completely unchanged between retries

## Workflow

1. Run the watchdog from the target repository root:

```bash
python /home/gguo/code/.codex/skills/slurm-self-healing-watchdog/scripts/slurm_self_heal.py \
  slurm/train.sbatch \
  --max-attempts 5
```

2. On each failed attempt, inspect the JSON report and the saved Codex repair note under the state directory.

3. If the job still fails after 5 attempts, stop and review the final classified reason.

## Useful Flags

- `--poll-seconds 30`
- `--state-dir logs/slurm/self-heal`
- `--codex-bin /home/gguo/.nvm/versions/node/v20.20.0/bin/codex`
- `--codex-model gpt-5.4`
- `--dry-run` to inspect derived paths without submitting

## Repair Policy

- The repair step should stay minimal and local to the repository.
- Prefer fixing the `sbatch` script, launcher, config, or the file directly implicated by the traceback.
- For `TIMEOUT` or `OUT_OF_MEMORY`, prefer resource, batch-size, checkpoint, or resume adjustments before broad code changes.
- If the failure is obviously transient, Codex may make no code changes and let the watchdog resubmit unchanged.
- If Codex cannot produce a safe repair, the watchdog stops instead of looping blindly.

## Output

Each run writes:
- one JSON report with attempt metadata
- one text file per repair attempt containing Codex's final summary

Read those artifacts first when deciding whether to continue manual debugging.