Fr Analysis

Analyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront process group where collectives diverge and returns the root-cause suspect ranks.

NVIDIA Updated 23k repo stars

File contents

nvidia/nvidia-resiliency-ext commit 2e2e03fd5b

Frequently asked questions

npx get-skill add nvidia/fr-analysis