Resubmit

Resubmit

Workflow

Use the bundled helper for repeatable job preparation:

python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect --failed-file status/failed.txt --reason "Q-Chem memory allocation / MEM_TOTAL too low"

Prefer running from the job directory that contains the .sh, .in, .out, and status/failed.txt files.

1. Identify target jobs.

   • Use --failed-file ... --reason ... to parse a reason section from failed.txt.
   • Or pass explicit .sh files after --scripts.
   • Confirm the number of target scripts before cleanup or submission.
   • The helper checks each corresponding .out file for successful Q-Chem completion and skips successful jobs.
   • The helper checks squeue for pending/running jobs with matching job names and skips jobs already in the queue.
   • If squeue cannot be queried, do not submit or run destructive cleanup; rerun with escalated permissions.

2. Check corresponding orbital folders.

   • The helper reads each .sh, finds QCSCRATCH, and parses the final scratch argument from the qchem command.
   • It reports how many scratch folders exist under /clusterfs/mhg-data/yaoshen/scf_read and prints one representative path.
   • It reports nested directories immediately inside target orbital folders and prints one representative nested path if any exist.

3. Remove stale copy commands, nested orbital folders, and qarchive.h5 files when requested or submitting.

   • Use --remove-existing-copy-lines during resubmission prep when scripts may contain orbital-folder copy commands.
   • The helper removes single-line cp or rsync commands whose destination is the target scratch/orbital folder, but only when that target folder already exists.
   • This prevents resubmitted jobs from copying old orbital folders over an existing target folder.
   • Use --remove-nested-orbitals only after the user grants permission to delete nested orbital directories.
   • The helper removes directories nested immediately inside each target orbital folder, not the target orbital folder itself.
   • Use --clean-qarchive during inspection only when stale orbital folders should be reused and the user has asked to clean them.
   • The helper removes files named qarchive.h5 inside existing target scratch folders.
   • Final submission with --submit always removes any existing qarchive.h5 files under submitted candidate scratch folders immediately before sbatch.
   • Do not delete whole scratch folders unless the user explicitly asks.

4. Check the .sh files.

   • The helper prints partition/account/QOS/memory distributions.
   • The helper queries sinfo for each used partition and checks that the maximum requested --mem is strictly smaller than the maximum node memory in that partition.
   • If Slurm queries are sandbox-blocked, rerun the helper with escalated permissions or run the sinfo check separately.
   • Do not change --mem during resubmission preparation unless the user explicitly asks.
   • If partition changes are needed, use the partition skill.

5. Print a representative .in file.

   • The helper prints the first representative input by default for read-only inspection.
   • Use --show-in-lines N to control preview length.

6. Submit only after permission.

   • Never run submission in the same step as initial inspection unless the user already gave explicit permission.
   • After the user approves, run the helper with --submit.
   • The helper submits only resubmission candidates, excluding successful outputs and active queued/running jobs.
   • The helper removes existing qarchive.h5 files under each submitted candidate's orbital folder before calling sbatch.
   • If sbatch is blocked by sandboxing, rerun with escalated permissions and explain that this submits jobs to Slurm.

Helper Examples

Inspect one failed reason section:

python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --failed-file /clusterfs/mhg-data/yaoshen/wb97m_os/GSCDB/revdod/status/failed.txt \
  --reason "Q-Chem memory allocation / MEM_TOTAL too low"

Inspect and remove stale qarchive files:

python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --failed-file /clusterfs/mhg-data/yaoshen/wb97m_os/GSCDB/revdod/status/failed.txt \
  --reason "Scratch/orbital/read-file corruption" \
  --remove-existing-copy-lines \
  --remove-nested-orbitals \
  --clean-qarchive

Submit after explicit approval:

python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --failed-file /clusterfs/mhg-data/yaoshen/wb97m_os/GSCDB/revdod/status/failed.txt \
  --reason "Q-Chem memory allocation / MEM_TOTAL too low" \
  --submit

Use explicit scripts instead of a failed report:

python3 /global/home/users/yaoshen/.codex/skills/resubmit/scripts/resubmit_helper.py inspect \
  --scripts job1.sh job2.sh

Safety Rules

• Ask before submitting jobs, even if the user asked to prepare resubmission.
• Skip resubmission for jobs whose .out files already show successful Q-Chem completion.
• Never modify, overwrite, truncate, delete, or regenerate .in or .out files. This skill may only read them for inspection, success detection, and preview.
• Skip resubmission for jobs already pending or running in squeue.
• Refuse submission if squeue cannot be checked.
• Treat --clean-qarchive and --submit as file deletion operations for existing qarchive.h5 files; --submit must remove these files from submitted candidate orbital folders before sbatch.
• Treat --remove-existing-copy-lines as a script edit operation; report how many lines and scripts changed.
• Treat --remove-nested-orbitals as a destructive directory deletion operation; ask the user first and report the representative nested path before deleting.
• Check the partition memory report before submission. Stop if any partition reports NOT OK.
• Report any missing .sh, .in, scratch folder, or qchem command before submitting.
• Stop before submission if any target script is missing, has no input file, or has no qchem command.