Fault Injection Loop

Skill: fault-injection-loop

Iterative closed-loop skill that runs a prioritized fault-injection experiment pool 2 jobs at a time, analyzes every artifact, scores attribution accuracy, aggregates gaps across the matrix, and proposes targeted improvements to attribution modules.

Overview

┌───────────────────────────────────────────────────────────────────────┐
│  0. POOL     → build ordered pool of (fault_type, rank, iter, nodes)  │
│               GPU faults first, then crash, Python-hang, signal       │
│                                                                        │
│  repeat until pool exhausted:                                          │
│  1. SUBMIT   → sbatch 2 jobs from pool head                            │
│  2. WAIT     → poll until both jobs leave RUNNING/PENDING              │
│                                                                        │
│  after all jobs done:                                                  │
│  3. ANALYZE  → watch_and_analyze.sh: /log-analysis + /fr-analysis     │
│               per completed job, streaming as jobs finish              │
│  4. SCORE    → compare attribution output vs injected ground truth     │
│  5. AGGREGATE→ build results table; identify systematic failure modes  │
│  6. IMPROVE  → patch restart_agent/ policy or evidence extraction      │
│  7. LOOP     → re-run same pool with updated attribution code          │
└───────────────────────────────────────────────────────────────────────┘

Step 0 — Fault Pool Design

The pool is defined as an ordered list of FAULT_TYPE:RANK:ITER:NODES entries inside scripts/prepare_node_alloc.sh. Default pool (34 experiments, 17 batches):

# GPU hangs — highest priority; full rank sweep across all node counts
GPU_SLEEP:1:5:2   GPU_SLEEP:0:5:2      # 2-node: rank-1, rank-0
GPU_SLEEP:4:5:2   GPU_SLEEP:7:5:2      # 2-node: mid-rank, last-rank
GPU_SLEEP:1:5:4   GPU_SLEEP:0:5:4      # 4-node: rank-1, rank-0
GPU_SLEEP:8:5:4   GPU_SLEEP:15:5:4     # 4-node: mid, last
GPU_SLEEP:1:5:8   GPU_SLEEP:0:5:8      # 8-node: rank-1, rank-0
GPU_SLEEP:16:5:8  GPU_SLEEP:31:5:8     # 8-node: mid, last

# GPU errors — high priority; rank-0 and rank-1 across all node counts
GPU_ERROR:1:5:2   GPU_ERROR:0:5:2
GPU_ERROR:1:5:4   GPU_ERROR:0:5:4
GPU_ERROR:1:5:8   GPU_ERROR:0:5:8

# Crash faults
SIGKILL:1:5:2     SIGKILL:0:5:2
SIGKILL:1:5:4     SIGKILL:1:5:8
SEGFAULT:1:5:2    SEGFAULT:0:5:2
SEGFAULT:1:5:4    OS_ABORT:1:5:2

# Python-level hangs
LOCK_GIL:1:5:2    LOCK_GIL:0:5:2
WORKLOAD_EXC:1:5:2  ASYNC_EXC:1:5:2

# Signals
SIGTERM:1:5:2     SIGINT:1:5:2
SIGSTOP:1:5:2     SIGNAL_EXC:1:5:2

Rank coverage per node count (4 GPUs/node):

To run a custom subset, override POOL before calling the script:

POOL="GPU_SLEEP:0:5:2 GPU_SLEEP:1:5:2" bash scripts/prepare_node_alloc.sh

Local User Config

Start from the tracked template:

cp scripts/user.env.example scripts/user.env

Then edit scripts/user.env with cluster-specific settings. This file is sourced by run_session.sh, prepare_node_alloc.sh, watch_and_analyze.sh, l4_gb200_reduced.sh, and n3_super_gb200_fi.sh. It is required for this skill to run and is intended to stay local and untracked.

Recommended contents:

PARTITION=gb-nvl-134-135
BASE_EXPERIMENTS_DIR="${HOME}/nvrx-attr-experiments"
MEGATRON_REPO_HOST_PATH="${HOME}/megatron-lm-main"
SHARED_TMP_BASE_DIR="${HOME}/tmp"
WORKSPACE_HOST_PATH="${HOME}/tmp"
CONTAINER_IMAGE="nvcr.io/nvidia/nemo:26.04"
LLM_API_KEY_FILE="${HOME}/.llm_api_key"
JUDGE_API_KEY_FILE="${HOME}/.llm_api_key"
NVRX_LLM_MODEL="nvidia/nemotron-3-super-120b-a12b"
NVRX_LLM_BASE_URL="https://integrate.api.nvidia.com/v1"
# RESTART_AGENT_CONFIG="/path/to/restart_agent.json"
JUDGE_MODEL="qwen/qwen3.5-397b-a17b"
JUDGE_BASE_URL="https://integrate.api.nvidia.com/v1"
FR_SEGMENT_SIZE=32

Use user.env for stable site defaults such as partition, container image, and host paths, plus local LLM credentials and endpoint settings for log-analysis and the judge. Use per-run environment overrides for experiment-specific controls such as POOL, WORKLOAD, BATCH_SIZE, FAULT_TYPE, FAULT_AT_ITER, or FAULT_DELAY.

If you use local Triton/Inductor cache staging, set the cache variables in scripts/user.env. See scripts/user.env.example for the supported ENABLE_NFS_CACHE_STAGING, NFS_TRITON_CACHE, and NFS_INDUCTOR_CACHE entries and workload-specific path examples.

Environment variables:

Registered workloads (scripts/workloads.conf)

Workload note:

• llama4_scout requires at least 2 nodes.
• n3_super requires at least 8 nodes. Its default registered pool contains only 8-node experiments.

# Run the full pool against the validated example workload
bash scripts/prepare_node_alloc.sh

# Run a custom subset against llama4_scout
POOL="GPU_SLEEP:1:5:2 SIGKILL:1:5:2" WORKLOAD=llama4_scout bash scripts/prepare_node_alloc.sh

Step 1 & 2 — Batched Submission + Wait (automated)

bash scripts/prepare_node_alloc.sh

The script loops: submit 2 jobs → poll squeue every 30 s until both finish → submit next 2. Progress is printed inline:

>>> Batch 1: experiments 1–2 of 34
  submitted: GPU_SLEEP rank=1  iter=5 nodes=2 -> job=1850
  submitted: GPU_SLEEP rank=0  iter=5 nodes=2 -> job=1851
  waiting for GPU_SLEEP:1:5:2 GPU_SLEEP:0:5:2 (1850,1851) ... 30s 60s done.
>>> Batch 2: experiments 3–4 of 34
  ...

A session directory and TSV tracking file are created at launch time:

${BASE_EXPERIMENTS_DIR}/fault_injection/<YYYYMMDD_HHMMSS>/
  experiments.tsv                              ← tracking file (all job IDs + paths)
  n<N>_<FAULT>_r<R>_i<I>/                     ← one subdir per experiment
    logs/slurm/<JOB_ID>.launch.out
    logs/slurm/<JOB_ID>.*.1.main_workload.log  ← log-analysis input
    checkpoints/                               ← fr-analysis input (FR dumps)
    tensorboard/
  experiments_report.md                        ← generated by watch_and_analyze.sh

Tracking file columns: JOB_ID FAULT_TYPE RANK ITER NODES EXPERIMENT_DIR

Step 3 — Analyze All Experiments

Run the watcher/analyzer — it reads the tracking file and processes each experiment as its job state leaves RUNNING/PENDING (works whether jobs are still running or already done):

bash scripts/watch_and_analyze.sh \
    ${BASE_EXPERIMENTS_DIR}/fault_injection/<YYYYMMDD_HHMMSS>/experiments.tsv

The watcher:

1. Reads each row from the tracking TSV
2. Calls restart-agent as python -m nvidia_resiliency_ext.attribution.restart_agent.cli and parses the JSON decision
3. Calls FR analysis as python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*" and passes the raw table output to the judge
4. Scores 7 dimensions (restart correctness, rank primary, rank any, category, type, FR rank)
5. Appends a scored row to <session>_report.md as a markdown table row
6. Repeats until all experiments are analyzed

To also run the sub-skills interactively for a single experiment:

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
  "${EXPERIMENT_DIR}/logs/slurm/${JOB_ID}.*.1.main_workload.log" --job-id "${JOB_ID}"
python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution \
  --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*"

Step 4 — Score Each Experiment

Scoring is performed by scripts/score_attribution.py, an LLM judge that receives the ground truth, the filtered raw log, the restart-agent attribution output, and the FR analysis output, then returns structured JSON scores with a reasoning note.

The judge is given:

1. Ground truth: fault_type, rank, iter, nodes
2. Expected restart decision + rationale (derived from score_attribution.py:_RESTART_TABLE)
3. Filtered raw log (last 400 lines, with fault-injection markers removed)
4. Raw restart-agent JSON output
5. Raw FR analysis table output from fr_attribution.py --fr-path ... -p "_dump_*"
6. GPUS_PER_NODE and FR_SEGMENT_SIZE to map the injected rank to exact node-sized and segment-sized scopes for FR scoring

Default judge model: qwen/qwen3.5-397b-a17b. Override with --model in score_attribution.py. Default segment size for FR scope scoring: 32 ranks. Override with FR_SEGMENT_SIZE.

Step 5 — Aggregate Results

The canonical output of the loop is the markdown table in <session>_report.md. When summarizing results for users, prefer linking to that file and reproducing the same table shape rather than flattening the results into plain prose.

The report markdown table from watch_and_analyze.sh gives a matrix view. Look for patterns across rows:

| FAULT_TYPE | NODES | RANK | restart_correct | rank_primary | rank_any | fault_described | fr_rank_correct | judge_notes |
|------------|-------|------|-----------------|--------------|----------|-----------------|-----------------|-------------|
| GPU_SLEEP  |   2   |  0   |      true       |    false     |   true   |      true       |      true       | rank-0 identified only in secondary issues |
| GPU_SLEEP  |   2   |  1   |      true       |     true     |   true   |      true       |      true       | correct on all dimensions |
| GPU_ERROR  |   2   |  1   |      false      |    false     |  false   |     partial     |      true       | LLM issued RESTART; rank not mentioned |
| SIGKILL    |   2   |  0   |      true       |    false     |  false   |     false       |      true       | attribution describes timeout not kill signal |

Common failure mode patterns and their meaning:

Step 6 — Identify and Apply Improvements

FR analysis

Deterministic graph algorithm — do not modify automatically. Note misidentifications and escalate to the team.

Log analysis (safe to modify)

Editable area: attribution/restart_agent/

After each patch, re-run the same pool subset that previously failed:

POOL="GPU_SLEEP:0:5:2 GPU_ERROR:1:5:2" bash scripts/prepare_node_alloc.sh

Step 7 — Loop

Increment experiment counter. Suggested sweep order across code-change iterations:

1. Iteration 1: full default pool (34 experiments)
2. Iteration 2: targeted re-run of all failing cells from iteration 1
3. Iteration 3: expand iter dimension (FAULT_AT_ITER=2 and 10) for remaining gaps
4. Iteration 4: add SEGFAULT and LOCK_GIL 4-node/8-node coverage

Stop condition: all cells pass all four scoring dimensions for two consecutive code-change iterations.

Adapting A SLURM Script For The Feedback Loop

The feedback loop is not tied to l4_gb200_reduced.sh, but your sbatch script must match a small contract so the loop can submit, analyze, and score each run.

Required changes for a custom workload script:

1. Accept these exported variables from prepare_node_alloc.sh: FAULT_TYPE, FAULT_RANK, FAULT_AT_ITER, EXPERIMENT_DIR, BASE_EXPERIMENTS_DIR, and GPUS_PER_NODE.
2. Write the main training log to: ${EXPERIMENT_DIR}/logs/slurm/${SLURM_JOB_ID}.*.1.main_workload.log so watch_and_analyze.sh can find it.
3. Write NCCL flight-recorder dumps under ${EXPERIMENT_DIR}/checkpoints/.
4. Emit a fault-injection marker when the fault is injected. watch_and_analyze.sh uses this to decide whether the run reached the injection point.
5. Preserve the per-experiment directory layout: logs/slurm/, checkpoints/, and tensorboard/.

This has only been validated with Megatron-LM because the current run-valid check and fault markers depend on Megatron's debug_fault_injection.py behavior. If you adapt the loop to another framework, update both the sbatch script and watch_and_analyze.sh.

Appendix A: SBATCH_SCRIPT fault parameters

The example SBATCH_SCRIPT reads these env vars from prepare_node_alloc.sh via --export:

Valid FAULT_TYPE values: GPU_ERROR, GPU_SLEEP, WORKLOAD_EXC, ASYNC_EXC, SIGNAL_EXC, OS_ABORT, LOCK_GIL, SEGFAULT, SIGINT, SIGKILL, SIGTERM, SIGSTOP

Appendix B: Single-experiment manual run

# Manual runs land under fault_injection/manual/ by default (no session dir needed)
EXPERIMENT_DIR=${HOME}/nvrx-attr-experiments/fault_injection/manual/n2_GPU_SLEEP_r1_i5
mkdir -p ${EXPERIMENT_DIR}/logs/slurm ${EXPERIMENT_DIR}/checkpoints ${EXPERIMENT_DIR}/tensorboard

sbatch \
    --nodes=2 \
    --output=${EXPERIMENT_DIR}/logs/slurm/%j.launch.out \
    --error=${EXPERIMENT_DIR}/logs/slurm/%j.launch.err \
    --export=ALL,FAULT_TYPE=GPU_SLEEP,FAULT_RANK=1,FAULT_AT_ITER=5,FAULT_DELAY=15,GPUS_PER_NODE=4,EXPERIMENT_DIR=${EXPERIMENT_DIR} \
    scripts/l4_gb200_reduced.sh

Optional site-specific cleanup:

export CONTAINER_CLEANUP_CMD='
ENROOT_DIR="/var/lib/enroot/data/$(id -u)"
rm -rf "${ENROOT_DIR:?}"/* 2>/dev/null || true
echo "$(hostname): / $(df -h / | tail -1 | awk "{print \$3\" used, \"\$4\" avail\"}")"
'