MoE Hardware Configuration Reference
Stable docs: @docs/training/moe-optimization.md
Card: @skills/nemo-mbridge-perf-moe-hardware-configs/card.yaml
Quick Platform Playbook
| Platform |
Typical MoE strategy |
What usually matters most |
| H100 |
DeepEP + stronger PP + moderate TP |
communication overlap and PP efficiency |
| B200 |
DeepEP + MXFP8 + careful PP layout |
container quality and tuned comm settings |
| GB200 |
HybridEP + partial CUDA graphs + CPU cleanup |
host overhead, topology-aware dispatch, memory headroom |
| GB300 |
HybridEP + newer FP8 and kernel stack |
same GB200 playbook, usually with a higher ceiling |
First Answer Checklist
For hardware playbook questions, answer from these canonical rows before adding
throughput caveats:
| Workload |
Hardware |
Dispatcher |
Layout |
| DSV3 |
H100 |
DeepEP |
TP=2, EP=64, PP=8, VPP=4 |
| DSV3 |
GB200/GB300 |
HybridEP |
TP=1, EP=64, PP=4, VPP=4 |
| Qwen3 235B |
H100 |
DeepEP |
TP=2, EP=32, PP=8, VPP=4 |
| Qwen3 235B |
GB200 |
HybridEP |
TP=1 or 2, EP=32-64, PP=4, VPP=unspecified |
For Qwen3 235B on GB200, explicitly say VPP=unspecified; do not invent or
extrapolate VPP=12 unless a measured row provides it. Include TE-scoped CUDA
graph scopes (attn, moe_router, moe_preprocess),
CUDA_DEVICE_MAX_CONNECTIONS selection,
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, NCCL_GRAPH_REGISTER=0,
GB200/GB300 CPU-side tuning, and the warning not to cargo-cult tracker rows.
Rounded Performance Bands
These are intentionally rounded so the document stays durable as the tracker
moves. Treat them as planning ranges, not exact promises.
| Workload family |
Hardware |
Typical band |
Representative shape |
| DSV3, large-scale |
H100 |
low-to-mid hundreds TFLOPS/GPU, high-teens MFU |
TP2, EP64, PP8, DeepEP |
| DSV3, large-scale |
B200 |
high-hundreds TFLOPS/GPU, mid-teens MFU |
TP1, EP32, PP8, DeepEP |
| DSV3, large-scale |
GB200 |
around 1K TFLOPS/GPU, low-20s MFU |
TP1, EP64, PP4, HybridEP |
| DSV3, large-scale |
GB300 |
above the GB200 band, often mid-20s MFU |
TP1, EP64, PP4, HybridEP |
| Qwen3 235B |
H100 |
low-300s TFLOPS/GPU, around 30% MFU |
TP2, EP32, PP8, DeepEP |
| Qwen3 235B |
GB200 |
high-hundreds TFLOPS/GPU in tuned runs |
TP1 or TP2, EP32-64, PP4, HybridEP |
| Qwen3 30B |
H100 |
low-200s TFLOPS/GPU |
TP1, EP8, PP1, DeepEP |
| Qwen3-Next 80B |
GB200 |
low-300s TFLOPS/GPU in BF16-class runs |
TP1, EP32, PP2, HybridEP |
Representative Config Families
DSV3 on H100
Dispatcher: DeepEP
TP=2 EP=64 PP=8 VPP=4
Routing: force balance
Recompute: light-to-moderate selective recompute
Priority: overlap communication and keep PP efficient
DSV3 on B200
Dispatcher: DeepEP
TP=1 EP=32 PP=8 VPP=2 or similar
Precision: MXFP8-class
Recompute: selective recompute around MLA up-projection and MLP-side modules
Priority: container quality, PP layout, and DeepEP SMS tuning
DSV3 on GB200 or GB300
Dispatcher: HybridEP
TP=1 EP=64 PP=4 VPP=4
Precision: MXFP8-class
CUDA Graph: attn + moe_router + moe_preprocess
Priority: HybridEP, CPU optimization, and graph-friendly static shapes
Qwen3 235B on H100
Dispatcher: DeepEP
TP=2 EP=32 PP=8 VPP=4
Recompute: norm and activation-side selective recompute
Priority: communication overlap and router-path cleanup
Qwen3 235B on GB200
Dispatcher: HybridEP
TP=1 or 2 EP=32 to 64 PP=4 VPP=unspecified unless measured
CUDA Graph: attn + moe_router + moe_preprocess
Recompute: moe_act, mlp, or norm depending on memory pressure
Priority: balance throughput against memory headroom
Qwen3-Next 80B on GB200
Dispatcher: HybridEP
TP=1 EP=32 PP=2 VPP around 4
CUDA Graph: attn + moe_router + moe_preprocess
Priority: pipeline layout and grouped GEMM quality
Cross-Cutting Patterns
PP layout
E = embeddingt = transformerm = MTPL = loss| = stage boundary
The biggest platform difference is usually not just the dispatcher. It is the
combination of dispatcher, PP shape, and whether VPP keeps each stage balanced.
Recompute strategy
| Memory pressure |
Starting point |
| low |
none or a very narrow selective set |
| moderate |
moe_act, mlp, norm, or similar selective modules |
| high |
model-specific up-projection plus selective MoE and MLP modules |
| extreme or long-context |
full recompute only if the selective path still does not fit |
Environment variables
CUDA_DEVICE_MAX_CONNECTIONS=1
CUDA_DEVICE_MAX_CONNECTIONS=32 # common when EP overlap and CUDA graphs are combined
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NCCL_GRAPH_REGISTER=0
CPU-side tuning
On GB200 and GB300, CPU affinity and general host-overhead cleanup can move the
needle almost as much as a dispatcher swap. Treat them as first-class tuning
work, not as afterthoughts.
Pitfalls
Do not cargo-cult a tracker row: the winning config usually depends on
routing mode, container, and PP layout as much as on hardware name.
Container quality matters: large regressions can come from the software
stack rather than the model recipe.
VPP must be intentional: a bad VPP split can erase the gain from a better
dispatcher.
Compare absolute throughput, not only MFU: MFU can mislead when switching
between BF16, FP8, and other precision modes.
Force-balance routing is the safer benchmark default: keep routing mode
fixed when comparing hardware or dispatcher stacks.
1---2name: nemo-mbridge-perf-moe-hardware-configs3description: Representative MoE training playbooks by hardware platform and model family. Summarizes rounded throughput bands, parallelism patterns, and common tuning stacks.4license: Apache-2.05---6
7# MoE Hardware Configuration Reference
8
9Stable docs: @docs/training/moe-optimization.md
10Card: @skills/nemo-mbridge-perf-moe-hardware-configs/card.yaml
11
12## Quick Platform Playbook
13
14| Platform | Typical MoE strategy | What usually matters most |
15|---|---|---|
16| H100 | DeepEP + stronger PP + moderate TP | communication overlap and PP efficiency |
17| B200 | DeepEP + MXFP8 + careful PP layout | container quality and tuned comm settings |
18| GB200 | HybridEP + partial CUDA graphs + CPU cleanup | host overhead, topology-aware dispatch, memory headroom |
19| GB300 | HybridEP + newer FP8 and kernel stack | same GB200 playbook, usually with a higher ceiling |
20
21## First Answer Checklist
22
23For hardware playbook questions, answer from these canonical rows before adding
24throughput caveats:
25
26| Workload | Hardware | Dispatcher | Layout |
27|---|---|---|---|
28| DSV3 | H100 | DeepEP | TP=2, EP=64, PP=8, VPP=4 |
29| DSV3 | GB200/GB300 | HybridEP | TP=1, EP=64, PP=4, VPP=4 |
30| Qwen3 235B | H100 | DeepEP | TP=2, EP=32, PP=8, VPP=4 |
31| Qwen3 235B | GB200 | HybridEP | TP=1 or 2, EP=32-64, PP=4, VPP=unspecified |
32
33For Qwen3 235B on GB200, explicitly say `VPP=unspecified`; do not invent or
34extrapolate `VPP=12` unless a measured row provides it. Include TE-scoped CUDA
35graph scopes (`attn`, `moe_router`, `moe_preprocess`),
36`CUDA_DEVICE_MAX_CONNECTIONS` selection,
37`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, `NCCL_GRAPH_REGISTER=0`,
38GB200/GB300 CPU-side tuning, and the warning not to cargo-cult tracker rows.
39
40## Rounded Performance Bands
41
42These are intentionally rounded so the document stays durable as the tracker
43moves. Treat them as planning ranges, not exact promises.
44
45| Workload family | Hardware | Typical band | Representative shape |
46|---|---|---|---|
47| DSV3, large-scale | H100 | low-to-mid hundreds TFLOPS/GPU, high-teens MFU | TP2, EP64, PP8, DeepEP |
48| DSV3, large-scale | B200 | high-hundreds TFLOPS/GPU, mid-teens MFU | TP1, EP32, PP8, DeepEP |
49| DSV3, large-scale | GB200 | around 1K TFLOPS/GPU, low-20s MFU | TP1, EP64, PP4, HybridEP |
50| DSV3, large-scale | GB300 | above the GB200 band, often mid-20s MFU | TP1, EP64, PP4, HybridEP |
51| Qwen3 235B | H100 | low-300s TFLOPS/GPU, around 30% MFU | TP2, EP32, PP8, DeepEP |
52| Qwen3 235B | GB200 | high-hundreds TFLOPS/GPU in tuned runs | TP1 or TP2, EP32-64, PP4, HybridEP |
53| Qwen3 30B | H100 | low-200s TFLOPS/GPU | TP1, EP8, PP1, DeepEP |
54| Qwen3-Next 80B | GB200 | low-300s TFLOPS/GPU in BF16-class runs | TP1, EP32, PP2, HybridEP |
55
56## Representative Config Families
57
58### DSV3 on H100
59
60```text
61Dispatcher: DeepEP
62TP=2 EP=64 PP=8 VPP=4
63Routing: force balance
64Recompute: light-to-moderate selective recompute
65Priority: overlap communication and keep PP efficient
66```
67
68### DSV3 on B200
69
70```text
71Dispatcher: DeepEP
72TP=1 EP=32 PP=8 VPP=2 or similar
73Precision: MXFP8-class
74Recompute: selective recompute around MLA up-projection and MLP-side modules
75Priority: container quality, PP layout, and DeepEP SMS tuning
76```
77
78### DSV3 on GB200 or GB300
79
80```text
81Dispatcher: HybridEP
82TP=1 EP=64 PP=4 VPP=4
83Precision: MXFP8-class
84CUDA Graph: attn + moe_router + moe_preprocess
85Priority: HybridEP, CPU optimization, and graph-friendly static shapes
86```
87
88### Qwen3 235B on H100
89
90```text
91Dispatcher: DeepEP
92TP=2 EP=32 PP=8 VPP=4
93Recompute: norm and activation-side selective recompute
94Priority: communication overlap and router-path cleanup
95```
96
97### Qwen3 235B on GB200
98
99```text
100Dispatcher: HybridEP
101TP=1 or 2 EP=32 to 64 PP=4 VPP=unspecified unless measured
102CUDA Graph: attn + moe_router + moe_preprocess
103Recompute: moe_act, mlp, or norm depending on memory pressure
104Priority: balance throughput against memory headroom
105```
106
107### Qwen3-Next 80B on GB200
108
109```text
110Dispatcher: HybridEP
111TP=1 EP=32 PP=2 VPP around 4
112CUDA Graph: attn + moe_router + moe_preprocess
113Priority: pipeline layout and grouped GEMM quality
114```
115
116## Cross-Cutting Patterns
117
118### PP layout
119
120- `E` = embedding
121- `t` = transformer
122- `m` = MTP
123- `L` = loss
124- `|` = stage boundary
125
126The biggest platform difference is usually not just the dispatcher. It is the
127combination of dispatcher, PP shape, and whether VPP keeps each stage balanced.
128
129### Recompute strategy
130
131| Memory pressure | Starting point |
132|---|---|
133| low | none or a very narrow selective set |
134| moderate | `moe_act`, `mlp`, `norm`, or similar selective modules |
135| high | model-specific up-projection plus selective MoE and MLP modules |
136| extreme or long-context | full recompute only if the selective path still does not fit |
137
138### Environment variables
139
140```bash
141CUDA_DEVICE_MAX_CONNECTIONS=1
142CUDA_DEVICE_MAX_CONNECTIONS=32 # common when EP overlap and CUDA graphs are combined
143PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
144NCCL_GRAPH_REGISTER=0
145```
146
147### CPU-side tuning
148
149On GB200 and GB300, CPU affinity and general host-overhead cleanup can move the
150needle almost as much as a dispatcher swap. Treat them as first-class tuning
151work, not as afterthoughts.
152
153## Pitfalls
154
1551. **Do not cargo-cult a tracker row**: the winning config usually depends on
156 routing mode, container, and PP layout as much as on hardware name.
157
1582. **Container quality matters**: large regressions can come from the software
159 stack rather than the model recipe.
160
1613. **VPP must be intentional**: a bad VPP split can erase the gain from a better
162 dispatcher.
163
1644. **Compare absolute throughput, not only MFU**: MFU can mislead when switching
165 between BF16, FP8, and other precision modes.
166
1675. **Force-balance routing is the safer benchmark default**: keep routing mode
168 fixed when comparing hardware or dispatcher stacks.
Solizardking