Fireworks AI Optimizes MiniMax M3 Sparse Attention Kernel for NVIDIA Blackwell, Achieving Significant Performance Boost
Summary:Fireworks AI has developed a KV-stationary sparse-attention kernel for the MiniMax M3 model, optimized for NVIDIA Blackwell (SM100) architecture. The kernel achieves approximately 980 TFLOP/s, outperforming the query-stationary baseline by 1.9–2.4× and open-source MSA by ~1.6×. The design leverages various optimizations, including contiguous partial-O stores with gathered combine, a 3-warp cp.async query gather, a shortened softmax→store pipeline, load-balanced split-Q scheduling, D2H eliminatio
Technical Highlights
-
KV-Stationary Sparse Attention Mechanism: Fireworks AI employs a KV-stationary sparse attention mechanism, which significantly improves computational efficiency compared to traditional query-stationary methods. This approach reduces redundant computations and optimizes memory access patterns, resulting in higher performance.
-
Multi-Level Optimizations:
- Contiguous Partial-O Stores with Gathered Combine: By storing partial outputs contiguously and using a gathered combine technique, the kernel reduces memory bandwidth requirements.
- 3-Warp cp.async Query Gather: Utilizes a 3-warp asynchronous mechanism to accelerate the collection of query data.
- Shortened Softmax→Store Pipeline: Shortens the pipeline from softmax to store, reducing latency.
- Load-Balanced Split-Q Scheduling: Employs a load-balanced query scheduling mechanism to enhance the utilization of overall computing resources.
- D2H Elimination: Reduces data transmission from device to host, further boosting efficiency.
- C++ AOT Dispatch Backend: Uses a C++-written AOT (Ahead-of-Time) dispatch backend to optimize task scheduling and resource management.
-
Performance: On the NVIDIA Blackwell architecture, the kernel achieves a peak performance of approximately 980 TFLOP/s, outperforming the baseline method by 1.9–2.4× and open-source MSA by ~1.6×.
Industry Impact and Recommendations for Developers
-
Impact on AI Model Optimization: The release of this kernel provides new insights into the optimization of AI models for sparse attention computations, particularly for applications requiring high efficiency and low latency, such as real-time inference and large-scale data processing.
-
Recommendations for NVIDIA Blackwell Users: NVIDIA Blackwell users can leverage Fireworks AI's optimized kernel to enhance the computational efficiency of their AI models. Developers can also draw inspiration from the kernel's design principles and optimization techniques for other AI model optimizations.
-
Future Research Directions: Future research could further explore the application of the KV-stationary sparse attention mechanism on other models and hardware platforms, as well as combine it with other optimization techniques, such as mixed-precision computing and model pruning, to achieve even greater performance improvements.
— END —Source: Fireworks AI Blog (2026-07-10)
Tags: #Fireworks AI #NVIDIA Blackwell #Sparse Attention #KV-Stationary #AI Optimization
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments