Prime Intellect Releases Prime Mega MoE, MoE Training Forward Speed Up by 3.11 Times

Techub News reports that AI infrastructure company Prime Intellect has released Prime Mega MoE, a trainable extension based on the DeepSeek open-source Mega MoE core. This core adds fast, overlapping backward propagation on top of DeepSeek's forward-only support, making it usable for training, not just inference.

In its prime-rl trainer, the Qwen3 MoE model's full training step speed increased by up to 34%. Compared to TorchTitan, the improvement reaches up to 55%. This core supports NVIDIA Hopper and Blackwell GPUs, utilizing BF16, MXFP8, and block FP8 data types.

Prime Intellect states that for teams training mixture of experts (MoE) models, this directly reduces the required GPU hours. Prime Mega MoE is a fused, overlapping MoE training core designed to address GPU idling caused by communication bottlenecks in MoE training. (MarkTechPost)