AWS Unveils MoE Kernel Optimizations: Enhancing Amazon SageMaker AI Inference Performance
Summary:AWS engineers have implemented kernel optimizations for fused Mixture-of-Experts (MoE) models on Amazon SageMaker AI, including Swap-AB matrix transposition for WGMMA utilization, Triton backend routing for FP8 quantization, and the elimination of Triton JIT recompilation. These optimizations result in up to a 10% increase in throughput and a 9% reduction in Time to First Token (TTFT) on H200 GPUs. This advancement highlights AWS's commitment to enhancing AI inference efficiency and provides cri
### Mixture-of-Experts (MoE) Kernel Optimizations Mixture-of-Experts (MoE) is a sparsely-activated architecture designed to scale the capacity of large language models (LLMs) while keeping per-token compute costs low. As its operations account for a significant portion of the total compute time, any efficiency gains directly reduce request latency and cost. Therefore, optimizing MoE operations through kernel engineering presents the largest opportunity for performance improvements. #### Swap-AB Matrix Transposition Optimization To maximize tensor processing throughput, the NVIDIA Hopper architecture relies on the Warp Group Matrix Multiply Accumulate (WGMMA) instruction protocol, which executes collective operations across a cooperative warpgroup of 128 threads. During the decode phase of MoE networks, the standard matrix layout leads to severe performance bottlenecks because the matrix A dimension is too small to fully utilize the hardware's fixed 64-row layout. By transposing the matrix multiplication, we realign the software to match the exact topography of the Hopper silicon. This transposition triggers two architectural advantages on the GPU: 1. **Physical Layout Saturation**: The expanded expert dimension (N=8192) is promoted to the first operand position (BT), and because N is large, it easily exceeds the hardware's mandatory 64-row threshold, allowing the Tensor Cores to operate at maximum local density with no wasted execution cycles. 2. **Asynchronous Prefetch Activation**: Hopper's dedicated hardware copy engines can asynchronously prefetch memory addresses from shared memory into registers before the math execution block fires. Placing the continuous, large weight blocks (BT) in the leading slot allows these engines to smoothly trace the data layout and keep the compute pipeline saturated without CPU intervention. This resulted in a 2-4% end-to-end output throughput improvement on H200 for large-scale MoE models. #### Triton Backend Routing for FP8 Quantization Beyond kernel internals, the choice of which backend executes the MoE computation has a significant impact on performance. The vLLM MoE oracle routes different quantization schemes to Triton, FlashInfer+CUTLASS, or other backends. Profiling revealed that for FP8 tensor-quantized models on Hopper, the Triton backend consistently outperforms the default FLASHINFER_CUTLASS path. The Triton backend excels in low-concurrency MoE workloads primarily because it consolidates the entire pipeline into a single hardware pass, eliminating slow global VRAM read/write roundtrips. This optimization yielded a 10% end-to-end throughput improvement for FP8 quantized fused MoE models on H200. #### Eliminating Triton JIT Recompilation One of the most impactful performance pathologies in GPU inference is unnecessary kernel recompilation. In MoE model serving, the _fused_inv_rope_fp8_quant_per_head kernel was recompiling on every new num_tokens value. Adding do_not_specialize annotations to the variable-length parameters removed these recompilations entirely, improving TTFT by 9% on H200 at 8-way tensor parallelism. In a multi-GPU cluster, eliminating recompilations across the entire cluster smoothed out the multi-GPU coordination overhead, delivering a 9% speedup for the user's initial response latency. The principle here is to treat the compiler as a first-class optimization surface, understanding how Triton specializes semantics and recognizing when parameters are dead or variable unlocks latency improvements with minimal code change. ### Conclusion Through our collaboration with the vLLM community, we implemented and open-sourced multiple kernel optimizations for fused MoE models. We apply these optimizations in the LMI Deep Learning Container on Amazon SageMaker AI. Together, these optimizations have delivered up to 10% higher throughput and up to 9% faster TTFT. --- > **Source**: [AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai/) (2026-10-13) — END —
Tags: #AWS #MoE #Kernel Optimization #AI Inference #vLLM
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments