Janus Moe Disaggregation

Skill Summary

Janus addresses MoE inference scaling through disaggregated architecture separating attention and MoE layers onto independent GPU sub-clusters. The system combines adaptive two-phase communication minimizing cross-cluster transfers, activation load-balanced scheduling distributing requests intelligently, and activation-aware expert management dynamically adjusting replication. Results show 3.9× higher per-GPU throughput while maintaining latency SLOs.

When To Use

• Deploying large sparse mixture-of-experts models with independent attention/expert scaling
• Scenarios where attention and MoE layers have different performance characteristics
• Projects with sufficient GPU clusters to justify disaggregation infrastructure
• Research on efficient MoE inference systems

When NOT To Use

• Small models where disaggregation overhead exceeds benefits
• Single-GPU or tightly coupled systems where disaggregation creates bottlenecks
• Latency-critical applications where communication overhead is problematic
• Scenarios with fixed hardware constraints preventing disaggregation

Core Technique

Four key components enable disaggregated MoE inference:

1. Disaggregated Architecture Rather than deploying entire MoE model as monolithic unit, manage attention and MoE layers separately. This enables fine-grained, module-specific resource scaling based on distinct performance characteristics.

2. Adaptive Two-Phase Communication Minimize overhead from frequent data transfers between attention and MoE instances:

• Intra-node aggregation via NVLink consolidates activations
• Bulk inter-node transfers reduce number of small cross-cluster messages Adaptive routing switches between phases based on activation characteristics.

3. Activation Load-Balanced Scheduling Lightweight GPU kernel scheduler distributes expert activation requests across MoE instances to minimize concurrently active experts per GPU. This reduces per-instance load and latency with negligible overhead.

4. Activation-Aware Expert Management Dynamically adjust expert replication counts and placement based on activation patterns. Spread frequently co-activated experts across GPUs to reduce per-instance load, improving throughput.

Implementation Notes

Design system separating attention and MoE layer clusters. Implement adaptive two-phase communication: intra-node via NVLink, inter-node in bulk transfers. Build activation load-balanced scheduler distributing requests intelligently. Track expert co-activation patterns and dynamically adjust replication. Monitor and optimize for your specific workload characteristics.

References

• Original paper: Janus (Dec 2025)
• Mixture-of-experts systems and inference
• Distributed GPU systems and communication optimization