# Resubmit

> Prepare and resubmit failed Slurm/Q-Chem jobs, especially benchmark jobs under wb97m_os/GSCDB. Use when Codex needs to collect failed job scripts, inspect matching orbital scratch folders under /clusterfs/mhg-data/yaoshen/scf_read, skip jobs whose .out files already succeeded, skip jobs already pending/running in Slurm, report or remove nested orbital folders, remove stale qarchive.h5 files, verify SBATCH partition/account/QOS/memory settings against current Slurm partition memory, preview representative .in files, and submit jobs only after explicit user permission.

- Skill: `alina-shen/resubmit` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add alina-shen/resubmit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/alina-shen/resubmit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: Alina-Shen (https://skillmd.com/u/alina-shen)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/alina-shen/resubmit

---


# Resubmit

## Workflow

Use the bundled helper for repeatable job preparation:

```bash
python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect --failed-file status/failed.txt --reason "Q-Chem memory allocation / MEM_TOTAL too low"
```

Prefer running from the job directory that contains the `.sh`, `.in`, `.out`, and `status/failed.txt` files.

1. Identify target jobs.
   - Use `--failed-file ... --reason ...` to parse a reason section from `failed.txt`.
   - Or pass explicit `.sh` files after `--scripts`.
   - Confirm the number of target scripts before cleanup or submission.
   - The helper checks each corresponding `.out` file for successful Q-Chem completion and skips successful jobs.
   - The helper checks `squeue` for pending/running jobs with matching job names and skips jobs already in the queue.
   - If `squeue` cannot be queried, do not submit or run destructive cleanup; rerun with escalated permissions.

2. Check corresponding orbital folders.
   - The helper reads each `.sh`, finds `QCSCRATCH`, and parses the final scratch argument from the `qchem` command.
   - It reports how many scratch folders exist under `/clusterfs/mhg-data/yaoshen/scf_read` and prints one representative path.
   - It reports nested directories immediately inside target orbital folders and prints one representative nested path if any exist.

3. Remove stale copy commands, nested orbital folders, and `qarchive.h5` files when requested or submitting.
   - Use `--remove-existing-copy-lines` during resubmission prep when scripts may contain orbital-folder copy commands.
   - The helper removes single-line `cp` or `rsync` commands whose destination is the target scratch/orbital folder, but only when that target folder already exists.
   - This prevents resubmitted jobs from copying old orbital folders over an existing target folder.
   - Use `--remove-nested-orbitals` only after the user grants permission to delete nested orbital directories.
   - The helper removes directories nested immediately inside each target orbital folder, not the target orbital folder itself.
   - Use `--clean-qarchive` during inspection only when stale orbital folders should be reused and the user has asked to clean them.
   - The helper removes files named `qarchive.h5` inside existing target scratch folders.
   - Final submission with `--submit` always removes any existing `qarchive.h5` files under submitted candidate scratch folders immediately before `sbatch`.
   - Do not delete whole scratch folders unless the user explicitly asks.

4. Check the `.sh` files.
   - The helper prints partition/account/QOS/memory distributions.
   - The helper queries `sinfo` for each used partition and checks that the maximum requested `--mem` is strictly smaller than the maximum node memory in that partition.
   - If Slurm queries are sandbox-blocked, rerun the helper with escalated permissions or run the `sinfo` check separately.
   - Do not change `--mem` during resubmission preparation unless the user explicitly asks.
   - If partition changes are needed, use the `partition` skill.

5. Print a representative `.in` file.
   - The helper prints the first representative input by default for read-only inspection.
   - Use `--show-in-lines N` to control preview length.

6. Submit only after permission.
   - Never run submission in the same step as initial inspection unless the user already gave explicit permission.
   - After the user approves, run the helper with `--submit`.
   - The helper submits only resubmission candidates, excluding successful outputs and active queued/running jobs.
   - The helper removes existing `qarchive.h5` files under each submitted candidate's orbital folder before calling `sbatch`.
   - If `sbatch` is blocked by sandboxing, rerun with escalated permissions and explain that this submits jobs to Slurm.

## Helper Examples

Inspect one failed reason section:

```bash
python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --failed-file /clusterfs/mhg-data/yaoshen/wb97m_os/GSCDB/revdod/status/failed.txt \
  --reason "Q-Chem memory allocation / MEM_TOTAL too low"
```

Inspect and remove stale qarchive files:

```bash
python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --failed-file /clusterfs/mhg-data/yaoshen/wb97m_os/GSCDB/revdod/status/failed.txt \
  --reason "Scratch/orbital/read-file corruption" \
  --remove-existing-copy-lines \
  --remove-nested-orbitals \
  --clean-qarchive
```

Submit after explicit approval:

```bash
python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --failed-file /clusterfs/mhg-data/yaoshen/wb97m_os/GSCDB/revdod/status/failed.txt \
  --reason "Q-Chem memory allocation / MEM_TOTAL too low" \
  --submit
```

Use explicit scripts instead of a failed report:

```bash
python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --scripts job1.sh job2.sh
```

## Safety Rules

- Ask before submitting jobs, even if the user asked to prepare resubmission.
- Skip resubmission for jobs whose `.out` files already show successful Q-Chem completion.
- Never modify, overwrite, truncate, delete, or regenerate `.in` or `.out` files. This skill may only read them for inspection, success detection, and preview.
- Skip resubmission for jobs already pending or running in `squeue`.
- Refuse submission if `squeue` cannot be checked.
- Treat `--clean-qarchive` and `--submit` as file deletion operations for existing `qarchive.h5` files; `--submit` must remove these files from submitted candidate orbital folders before `sbatch`.
- Treat `--remove-existing-copy-lines` as a script edit operation; report how many lines and scripts changed.
- Treat `--remove-nested-orbitals` as a destructive directory deletion operation; ask the user first and report the representative nested path before deleting.
- Check the partition memory report before submission. Stop if any partition reports `NOT OK`.
- Report any missing `.sh`, `.in`, scratch folder, or `qchem` command before submitting.
- Stop before submission if any target script is missing, has no input file, or has no `qchem` command.