LLMs1 min read
NVIDIA Transformer Engine Accelerates Dropless MoE Training
The NVIDIA Transformer Engine, when combined with JAX, delivers a 10.4x throughput improvement for Mixture of Experts training on NVIDIA GB200 GPUs, enabling DeepSeek-V3 to reach 1,068 TFLOPS/GPU. This achieves dropless MoE training by optimizing grouped GEMM kernels and NCCL EP for variable expert token counts.
From NVIDIA technical blog
