Betelgeuse Engine Achieves 24 tok/s on IQ4_XS Quantized Inference: Optimizing Memory Architecture for Enhanced LLM Effic
By Mr.Xu Community Post
Published:
Summary:The Betelgeuse team optimized the DMA path for the host and victim cache, achieving a 24 tok/s inference speed on the IQ4_XS quantized model using an RTX 3060 12GB GPU. This represents a 71.4% improvement over the single-channel memory configuration. The results highlight the potential of dual-channel DDR4 memory and host caching mechanisms to enhance LLM inference efficiency, particularly in resource-constrained environments.
Technical Mechanism Analysis
The Betelgeuse Engine achieves high inference performance through the following technical mechanisms:
-
Exclusive Host RAM Victim Cache: When an expert is evicted from VRAM, it is stored in the host RAM victim cache. This mechanism ensures that VRAM and RAM do not store duplicate data, optimizing memory usage.
-
Async Hardware DMA: By using asynchronous hardware DMA, Betelgeuse can efficiently transfer data between VRAM and the NVMe drive. This approach reduces competition between the CPU and the PCIe H2D DMA engine, improving data transfer speed.
-
MOE_DMA=1 Configuration: This configuration enables the DMA function for the Mixture of Experts (MoE) model, further optimizing the data transfer path.
Engineering Trade-offs and Performance
-
Impact of Memory Bandwidth: In the single-channel DDR4 (16GB) configuration, the inference speed of IQ4_XS is only 14 tok/s because the single-channel memory bandwidth is limited to about 17-19 GB/s. In contrast, in the dual-channel DDR4 (2x 8GB) configuration, the memory bandwidth doubles to 38 GB/s, boosting the inference speed to 24 tok/s.
-
Quantization Precision and Performance: Although the expert size of IQ4_XS is almost twice that of IQ2_XS, by optimizing the memory architecture, Betelgeuse achieves higher inference speed while maintaining 100% byte-level accuracy.
-
Hardware Limitations and Optimization: In resource-constrained hardware environments (such as RTX 3060 12GB), by optimizing memory usage and DMA paths, Betelgeuse demonstrates the possibility of achieving efficient inference on low-power hardware.
Developer Implementation and Deployment Recommendations
-
Memory Configuration Optimization: It is recommended to use dual-channel DDR4 memory when deploying the Betelgeuse Engine to fully utilize its memory bandwidth optimization features.
-
Hardware Selection: Although Betelgeuse performs well on the RTX 3060 12GB, for applications requiring higher inference speed, it is recommended to use higher-performance GPUs and faster memory configurations.
-
Software Optimization: Developers can refer to the open-source code repository of Betelgeuse (GitHub) for further optimization and customization.
Future Outlook
The Betelgeuse team is applying this streaming and victim cache architecture to the DeepSeek V4.1 Flash (1017) model. Early tests show that the inference speed can reach about 25 tok/s on the same hardware. This indicates that the architecture has broad applicability and potential for further optimization.
Conclusion
The Betelgeuse Engine, through innovative memory architecture and DMA optimization, achieves efficient quantized model inference in resource-constrained hardware environments. This achievement not only demonstrates the application potential of AI models on low-power hardware but also provides new optimization ideas for AI developers.
— END —Source: Reddit r/LocalLLaMA (2026-10-31)
Tags: #Betelgeuse #LLM Inference #Memory Optimization #Quantized Models #AI Architecture
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments