arXiv Introduces New Linear-Time Transformer Method via Taylor Series Expansion and Fixed-State Recurrence

Summary:arXiv has introduced a novel linear-time Transformer architecture based on Taylor series expansion. By expanding the exponential kernel of attention into a truncated series and factorizing each term into tensor-power inner products, the method collapses the causal history into a fixed number of recurrent states. This approach significantly reduces computational complexity while maintaining accuracy, particularly excelling in long-sequence tasks. Experiments show that the method is 20.8 times fas

---

Technical Mechanism Analysis

The study proposes a novel linear-time Transformer architecture based on Taylor series expansion. The core idea is to expand the exponential kernel of attention into a polynomial series and factorize each term into tensor-power inner products. Specifically, the causal history is collapsed into a fixed number of recurrent states (n states), each updated by a rank-1 outer product and independent of the context length N. This approach has the following characteristics:

1. Recursion Losslessness: The recursion is lossless with respect to the truncated kernel, with a precision loss of $2.2 \times 10^{-16}$ (in float64).
2. State Additivity: The states are additive across chunks, with a precision loss of $5.7 \times 10^{-14}$, allowing for parallelization of the prefill operation.
3. Performance Improvement: In long-sequence tasks, the state update speed scales with $N^{0.91}$, while traditional Softmax scales with $N^{2.28}$. At $N = 8192$, the state update is 20.8 times faster than Softmax.
4. Decoding Latency: The per-token decoding latency is constant at $7.3\mu$s, while Softmax's latency scales with $N^{1.04}$, resulting in a 76-fold speedup at $N = 65536$.

Engineering Trade-offs and Performance

While the method excels in long-sequence tasks, there are some trade-offs:

• Precision vs. Cost: Precision improves exponentially with n, but the cost grows combinatorially because the m-th state has a width of $\binom{d+m-1}{m}$. This is confirmed by SVD, making the optimal choice second-order (n=3), with an error of $6.5 \times 10^{-4}$ and 9945 elements per head.
• Normalization Requirement: Query and key normalization is necessary; otherwise, the Taylor series argument exceeds the convergence radius, and adding more terms does not help.

Developer Deployment and Implementation Recommendations

1. Applicable Scenarios: This method is particularly suitable for tasks that require processing long sequences, such as long-document translation, speech recognition, and video processing.
2. Optimization Tips: It is recommended to prioritize second-order expansion (n=3) in resource-constrained environments to balance precision and computational cost.
3. Future Directions: Future research could further explore how to dynamically adjust the number of recurrent states to adapt to different task requirements and resource constraints.

Conclusion

This research provides a new approach to the linear-time implementation of Transformer architectures, showing great potential in long-sequence tasks. By using Taylor series expansion and fixed-state recurrence, the method significantly reduces computational complexity while maintaining high precision, laying the foundation for the further development of AI models.

---

Source: ArXiv Machine Learning (cs.LG) (2026-10-16)

— END —

Tags: #Transformer Follow #Taylor Series Follow #Linear-Time Follow #Long-Sequence Processing Follow #arXiv Follow