HPC Runtime Doctor
Goal
Turn cluster symptoms into a resource-layout diagnosis, environment checklist, and safe retry plan.
Requirements
- Python 3.10+
- No external dependencies
- Works on Linux, macOS, and Windows
Inputs to Gather
| Input | Description | Example |
|---|---|---|
| Scheduler | SLURM, PBS, LSF, local | slurm |
| Nodes/tasks/threads | Runtime layout | 2 nodes, 128 tasks, 2 threads |
| GPUs | GPUs requested | 4 |
| Symptoms | Observed failure | oom,killed,slow-gpu |
| MPI/OpenMP/GPU use | Parallel modes | mpi+openmp+gpu |
| Walltime | Requested time | 12:00:00 |
| Scratch | Whether scratch is used | true |
Decision Guidance
- Check resource layout before changing physics settings.
- Confirm module/compiler/MPI/CUDA consistency before debugging solver behavior.
- Treat missing restart files and scratch cleanup as workflow failures, not physics failures.
- For GPU jobs, confirm the executable was built with the requested accelerator backend.
Script Outputs
scripts/hpc_runtime_doctor.py emits:
resource_layoutdiagnosesenvironment_checksretry_planscheduler_notes
Workflow
python3 skills/hpc-deployment/hpc-runtime-doctor/scripts/hpc_runtime_doctor.py \
--scheduler slurm \
--nodes 2 \
--tasks 128 \
--cpus-per-task 2 \
--gpus 4 \
--symptoms oom,slow-gpu \
--uses-mpi \
--uses-openmp \
--uses-gpu \
--json
Error Handling
Invalid resource counts stop with exit code 2. Unknown symptoms are preserved as custom items for human review.
Limitations
This skill does not query a live scheduler. It diagnoses from the submitted layout and symptoms.
Security
- Inputs are scalar CLI values and booleans only.
- The script does not execute scheduler commands or inspect environment variables.
- The skill uses
Bashonly to run its bundled script.
References
- See
references/hpc_runtime_patterns.mdfor scheduler and runtime diagnosis patterns.
Version History
- 1.0.0: Initial HPC runtime diagnosis skill.