ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #LLMs & Foundation Models #GPU Optimization #Transcendental Functions #bfloat16 #Training Efficiency

arXiv Proposes Fast Polynomial Transcendentals to Enhance LLM Training Efficiency

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 10 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A new arXiv paper proposes using short polynomial programs to accelerate special function unit (SFU) operations, such as sigmoid, tanh, and SiLU, in large language models (LLMs) on GPUs. The research demonstrates that replacing native PyTorch evaluations with bfloat16 (BF16) polynomial programs in tasks like dense-SiLU, tanh-softcapped attention, and routed-expert SwiGLU can lead to notable throughput improvements, such as a 2.7% increase for dense-SiLU and an 8.0% increase for routed-expert Swi


Background and Motivation

In recent years, the rapid development of large language models (LLMs) has led to an increasing demand for computational resources during training and inference. While GPU hardware has made significant strides in matrix operations and memory access, bottlenecks still exist in handling special function unit (SFU) operations. This research aims to enhance LLM training efficiency by optimizing the computation of transcendental functions such as sigmoid, tanh, and SiLU.

Methodology and Implementation

The research team proposes using short polynomial programs to replace native PyTorch evaluations with bfloat16 (BF16) polynomial programs. Specifically, they designed degree-3 and degree-4 BF16 polynomial programs and applied them to four GB200 integration tasks: dense SiLU, tanh-softcapped attention, sigmoid attention, and routed-expert Swish-gated linear unit (SwiGLU).

Key Results

  1. Performance Improvements: The improved paths achieved a 1.19 to 2.19x speedup in L2 cache and a 1.00 to 1.70x speedup in high-bandwidth memory (HBM).
  2. Training Efficiency: The throughput of dense-SiLU increased by 2.7%, tanh-softcapped attention by 2.9%, and routed-expert SwiGLU by 8.0% in complete training steps. The forward computation of sigmoid-attention improved by 7.4%, while the complete GPU step improved by only 0.3%.
  3. Model Behavior Impact: The final smoothed training loss differences (polynomial minus native) across the four tasks ranged from -0.107 to +0.079 at the 100 billion token mark, indicating that the optimizations have a minimal impact on model behavior.

Technical Highlights

  • Transcendental Function Optimization: The use of polynomial programs to accelerate transcendental function computations addresses the GPU bottleneck in SFU operations.
  • bfloat16 Precision: The use of bfloat16 precision ensures computational accuracy while significantly improving computational efficiency.
  • Integration Task Validation: The effectiveness of the method is validated across multiple integration tasks, ensuring its feasibility in real-world applications.

Industry Impact and Developer Recommendations

This research provides a new approach to improving LLM training efficiency, particularly in handling complex transcendental functions. The results demonstrate that the optimization method enhances training speed while maintaining model performance. For developers, it is advisable to consider incorporating similar polynomial program optimizations into the LLM training pipeline to boost overall efficiency. Additionally, this research offers insights for future GPU hardware design, highlighting the importance of further optimizing SFU operations at the hardware level.


Source: ArXiv Machine Learning (cs.LG) (2026-10-02)

— END —

Tags: #LLMs & Foundation Models #GPU Optimization #Transcendental Functions #bfloat16 #Training Efficiency

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…