Slurm Self Healing Watchdog

Slurm Self Healing Watchdog

Use this skill when a Slurm job should not stop after one failed sbatch.

What It Does

The bundled script:

• submits an sbatch script
• polls sacct and falls back to squeue
• resolves stdout/stderr paths from #SBATCH -o/-e
• classifies the failure mode from Slurm state plus log text
• calls codex exec with a focused repair prompt when the job did not complete
• resubmits the job after repair, up to --max-attempts 5
• writes a JSON report for all attempts and repair summaries

When To Use

Use this skill when the user wants:

• a post-submit trigger that watches whether a Slurm job stopped
• automatic retry after timeout, OOM, node failure, launcher failure, or deterministic script bugs
• Codex to inspect logs and make a minimal fix before resubmission
• a reusable wrapper around sbatch rather than a one-off manual loop

Do not use it when:

• the user only wants a plain one-shot submit
• the failure requires human judgment beyond safe local edits
• the repository must remain completely unchanged between retries

Workflow

1. Run the watchdog from the target repository root:

python /home/gguo/code/.codex/skills/slurm-self-healing-watchdog/scripts/slurm_self_heal.py \
  slurm/train.sbatch \
  --max-attempts 5

2. On each failed attempt, inspect the JSON report and the saved Codex repair note under the state directory.

3. If the job still fails after 5 attempts, stop and review the final classified reason.

Useful Flags

• --poll-seconds 30
• --state-dir logs/slurm/self-heal
• --codex-bin /home/gguo/.nvm/versions/node/v20.20.0/bin/codex
• --codex-model gpt-5.4
• --dry-run to inspect derived paths without submitting

Repair Policy

• The repair step should stay minimal and local to the repository.
• Prefer fixing the sbatch script, launcher, config, or the file directly implicated by the traceback.
• For TIMEOUT or OUT_OF_MEMORY, prefer resource, batch-size, checkpoint, or resume adjustments before broad code changes.
• If the failure is obviously transient, Codex may make no code changes and let the watchdog resubmit unchanged.
• If Codex cannot produce a safe repair, the watchdog stops instead of looping blindly.

Output

Each run writes:

• one JSON report with attempt metadata
• one text file per repair attempt containing Codex's final summary

Read those artifacts first when deciding whether to continue manual debugging.