Elastic Fabric Adapter (EFA) Diagnostics
When to use
Any EFA investigation — creation failures, attachment errors, driver installation, libfabric issues, MPI communication errors, NCCL failures, performance problems, bandwidth limitations, security group configuration, placement groups, or SageMaker integration.
Investigation workflow
Step 1 — Collect and triage
aws ec2 describe-instances --instance-ids <instance-id> --query 'Reservations[0].Instances[0].NetworkInterfaces[*].{Id:NetworkInterfaceId,Type:InterfaceType,Status:Status}'
aws ec2 describe-network-interfaces --filters "Name=interface-type,Values=efa" --query 'NetworkInterfaces[*].{Id:NetworkInterfaceId,InstanceId:Attachment.InstanceId,Status:Status,AZ:AvailabilityZone}'
aws ec2 describe-instance-types --instance-types <type> --query 'InstanceTypes[0].{EfaSupported:NetworkInfo.EfaSupported,MaxEfa:NetworkInfo.EfaInfo.MaximumEfaInterfaces}'
Step 2 — Domain deep dive
fi_info -p efa # List EFA provider capabilities
fi_pingpong -p efa # Test basic EFA connectivity between nodes
ibv_devinfo # Show EFA device information and capabilities
Step 3 — Detailed investigation
aws ec2 describe-security-groups --group-ids <sg-id> --query 'SecurityGroups[0].{Inbound:IpPermissions,Outbound:IpPermissionsEgress}'
aws ec2 describe-placement-groups --group-names <group> --query 'PlacementGroups[0].{Strategy:Strategy,State:State,GroupId:GroupId}'
dmesg | grep -i efa # Check kernel messages for EFA driver issues
Read references/guardrails.md before concluding on any EFA issue.
Tool quick reference
| Tool / API |
When to use |
ec2 describe-network-interfaces |
List EFA interfaces and status |
ec2 describe-instance-types |
Check EFA support for instance type |
fi_info -p efa |
Verify libfabric EFA provider |
fi_pingpong -p efa |
Test EFA connectivity |
ibv_devinfo |
Check EFA device capabilities |
ec2 describe-security-groups |
Verify EFA security group rules |
ec2 describe-placement-groups |
Check placement group configuration |
Gotchas: Elastic Fabric Adapter
- EFA is only supported on specific instance types (p4d, p5, trn1, hpc6a, hpc7g, c5n.18xlarge, etc.). Not all large instances support EFA.
- EFA requires a security group that allows ALL traffic from itself (self-referencing rule). Without this, EFA traffic is blocked.
- EFA uses OS-bypass for kernel-bypass networking. The EFA driver and libfabric must be installed on the instance — they are NOT included in standard AMIs.
- Placement groups with cluster strategy are STRONGLY recommended for EFA. Without them, instances may be placed across different network switches, degrading performance.
- EFA and ENA (Elastic Network Adapter) are different. EFA provides additional OS-bypass capabilities on top of ENA. An EFA interface also functions as an ENA interface.
- NCCL (NVIDIA Collective Communications Library) requires the aws-ofi-nccl plugin to use EFA. Without it, NCCL falls back to TCP sockets.
- EFA interfaces cannot be attached to running instances — they must be specified at launch time or attached when the instance is stopped.
Anti-hallucination rules
- Always cite specific instance types, interface IDs, or driver versions as evidence.
- EFA support is instance-type specific. Never assume an instance type supports EFA without checking.
- EFA requires OS-bypass drivers. Never suggest EFA works with standard networking drivers alone.
- Security group self-referencing rules are mandatory. Never skip this requirement.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
12 runbooks
| Category |
IDs |
Covers |
| A — Creation & Attachment |
A1–A2 |
EFA creation failures, attachment errors |
| B — Drivers & Libraries |
B1–B2 |
Driver installation, libfabric issues |
| C — Communication |
C1–C2 |
MPI communication errors, NCCL errors |
| D — Performance |
D1–D2 |
Performance issues, bandwidth limitations |
| E — Network Config |
E1–E2 |
Security group configuration, placement groups |
| F — Integration |
F1 |
SageMaker integration |
| Z — Catch-All |
Z1 |
General troubleshooting |
1---2name: efa-diagnostics3description: Use this skill to investigate and troubleshoot Elastic Fabric Adapter (EFA) problems by analyzing EFA creation, attachment, driver installation, libfabric, MPI communication, NCCL, performance, bandwidth, security groups, placement groups, SageMaker integration, and following structured runbooks. Activate when: EFA creation failures, attachment errors, driver issues, libfabric problems, MPI errors, NCCL failures, performance degradation, bandwidth limitations, security group misconfigurations, placement group issues, SageMaker EFA problems, or the user says something is wrong with EFA.4---56# Elastic Fabric Adapter (EFA) Diagnostics78## When to use910Any EFA investigation — creation failures, attachment errors, driver installation, libfabric issues, MPI communication errors, NCCL failures, performance problems, bandwidth limitations, security group configuration, placement groups, or SageMaker integration.1112## Investigation workflow1314### Step 1 — Collect and triage1516```17aws ec2 describe-instances --instance-ids <instance-id> --query 'Reservations[0].Instances[0].NetworkInterfaces[*].{Id:NetworkInterfaceId,Type:InterfaceType,Status:Status}'18aws ec2 describe-network-interfaces --filters "Name=interface-type,Values=efa" --query 'NetworkInterfaces[*].{Id:NetworkInterfaceId,InstanceId:Attachment.InstanceId,Status:Status,AZ:AvailabilityZone}'19aws ec2 describe-instance-types --instance-types <type> --query 'InstanceTypes[0].{EfaSupported:NetworkInfo.EfaSupported,MaxEfa:NetworkInfo.EfaInfo.MaximumEfaInterfaces}'20```2122### Step 2 — Domain deep dive2324```25fi_info -p efa # List EFA provider capabilities26fi_pingpong -p efa # Test basic EFA connectivity between nodes27ibv_devinfo # Show EFA device information and capabilities28```2930### Step 3 — Detailed investigation3132```33aws ec2 describe-security-groups --group-ids <sg-id> --query 'SecurityGroups[0].{Inbound:IpPermissions,Outbound:IpPermissionsEgress}'34aws ec2 describe-placement-groups --group-names <group> --query 'PlacementGroups[0].{Strategy:Strategy,State:State,GroupId:GroupId}'35dmesg | grep -i efa # Check kernel messages for EFA driver issues36```3738Read `references/guardrails.md` before concluding on any EFA issue.3940## Tool quick reference4142| Tool / API | When to use |43|------------|-------------|44| `ec2 describe-network-interfaces` | List EFA interfaces and status |45| `ec2 describe-instance-types` | Check EFA support for instance type |46| `fi_info -p efa` | Verify libfabric EFA provider |47| `fi_pingpong -p efa` | Test EFA connectivity |48| `ibv_devinfo` | Check EFA device capabilities |49| `ec2 describe-security-groups` | Verify EFA security group rules |50| `ec2 describe-placement-groups` | Check placement group configuration |5152## Gotchas: Elastic Fabric Adapter5354- EFA is only supported on specific instance types (p4d, p5, trn1, hpc6a, hpc7g, c5n.18xlarge, etc.). Not all large instances support EFA.55- EFA requires a security group that allows ALL traffic from itself (self-referencing rule). Without this, EFA traffic is blocked.56- EFA uses OS-bypass for kernel-bypass networking. The EFA driver and libfabric must be installed on the instance — they are NOT included in standard AMIs.57- Placement groups with cluster strategy are STRONGLY recommended for EFA. Without them, instances may be placed across different network switches, degrading performance.58- EFA and ENA (Elastic Network Adapter) are different. EFA provides additional OS-bypass capabilities on top of ENA. An EFA interface also functions as an ENA interface.59- NCCL (NVIDIA Collective Communications Library) requires the aws-ofi-nccl plugin to use EFA. Without it, NCCL falls back to TCP sockets.60- EFA interfaces cannot be attached to running instances — they must be specified at launch time or attached when the instance is stopped.6162## Anti-hallucination rules63641. Always cite specific instance types, interface IDs, or driver versions as evidence.652. EFA support is instance-type specific. Never assume an instance type supports EFA without checking.663. EFA requires OS-bypass drivers. Never suggest EFA works with standard networking drivers alone.674. Security group self-referencing rules are mandatory. Never skip this requirement.685. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.6970## 12 runbooks7172| Category | IDs | Covers |73|----------|-----|--------|74| A — Creation & Attachment | A1–A2 | EFA creation failures, attachment errors |75| B — Drivers & Libraries | B1–B2 | Driver installation, libfabric issues |76| C — Communication | C1–C2 | MPI communication errors, NCCL errors |77| D — Performance | D1–D2 | Performance issues, bandwidth limitations |78| E — Network Config | E1–E2 | Security group configuration, placement groups |79| F — Integration | F1 | SageMaker integration |80| Z — Catch-All | Z1 | General troubleshooting |
aws-samples