Scaling large language models with Mixture-of-Experts (MoE) architecture involves complex infrastructure. MoE models can reach hundreds of billions or trillions of parameters while maintaining efficient inference through sparsity.
Training these models at scale requires managing multiple workloads. These include pre-training, supervised fine-tuning, and reinforcement learning (RL). Large-scale RL adds demands for high-bandwidth communication and elastic inference work.
When training MoE models with reinforcement learning, communication overhead becomes a key challenge. Expert Parallelism (EP) routes tokens across devices dynamically, increasing inter-node traffic as models grow larger.
AWS uses Amazon Elastic Kubernetes Service (EKS), Elastic Fabric Adapter (EFA), and DeepEP to address these issues. DeepEP improves expert-parallel communication over EFA, boosting throughput by 40%.
Training workloads must balance rollout generation and policy training. Rollout focuses on maximizing throughput, while policy training requires tightly synchronized workers.
As training scales beyond a single node, communication shifts from high-bandwidth intra-node NVLink to lower-bandwidth inter-node links. MoE models intensify this shift due to dynamic token routing.
Why it matters
This architecture helps run large-scale reinforcement learning more efficiently, saving time and costs.
What to do
Consider using Amazon EKS, EFA, and DeepEP for large-scale RL training. Optimize workload balancing to prevent bottlenecks.
