Fault Injection Loop

Closed-loop fault injection and attribution accuracy benchmark. Draws from a prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them 2 at a time via sbatch — waiting for each pair to finish before submitting the next — to bound filesystem load. GPU-related faults are front-loaded in the pool. After all jobs complete, runs /log-analysis and /fr-analysis on every experiment, scores attribution vs. ground truth, aggregates gaps, and iterates on attribution modules to close them.

NVIDIA Updated 23k repo stars

File contents

nvidia/nvidia-resiliency-ext commit bdf840a38d

Frequently asked questions

npx get-skill add nvidia/fault-injection-loop