ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #Sparse Transformer #Continual Learning #Edge AI #Low-Rank Adaptation #Parameter Efficiency

ArXiv Introduces sLoTh: Revolutionizing Continual Learning for Sparse Event-Driven Transformers

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 2 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:The ArXiv team introduces sLoTh, a novel framework designed to address the challenges of continual learning for sparse event-driven (spiking) vision Transformers. By freezing the model backbone and restricting updates to scalable-efficient low-rank attention (seLoRA) and shared neuronal threshold modulation, sLoTh enables parameter-efficient adaptation with less than 1% of model parameters being updated. Experiments across multiple datasets demonstrate competitive performance in class-incrementa


Background and Challenges

In dynamic environments, robotic and edge intelligence systems need to continuously adapt to new data while retaining previously learned knowledge, posing stringent requirements on the continual learning capabilities of models. However, traditional vision Transformers rely on dense computation, making them costly for real-world deployment. Sparse event-driven vision Transformers offer efficient event-based computation but their continual learning capabilities remain largely unexplored.

Overview of sLoTh Framework

sLoTh is a parameter-efficient continual learning framework for pretrained sparse event-driven (spiking) vision Transformers. Its key innovations include:

  1. Backbone Freezing and Low-Rank Updates: By freezing the model backbone and allowing only low-rank attention updates (seLoRA) for parameter adjustments.
  2. Shared Neuronal Threshold Modulation: Introducing a shared neuronal threshold modulation mechanism to further reduce the number of parameters that need to be updated.
  3. Replay-Free Learning: Achieving continual learning without replay buffers by updating less than 1% of the model parameters.

Experimental Results

sLoTh was tested on benchmark datasets such as CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-R, covering up to 100 tasks. The results demonstrate that:

  • sLoTh achieves competitive performance in class-incremental and online continual learning.
  • Energy consumption is reduced by approximately 6.5 times compared to traditional dense vision Transformers.

Technical Highlights

  • Parameter Efficiency: By limiting the number of updated parameters, it significantly reduces computational and storage costs.
  • Low Energy Consumption: While maintaining high performance, it greatly reduces energy consumption, making it suitable for edge device deployment.
  • Flexibility: The framework is designed to be flexible and can be extended to other types of Transformer architectures.

Industry Impact

The sLoTh framework provides a new solution for continual learning in edge AI and robotics applications. Its high efficiency and low energy consumption make it widely applicable in resource-constrained environments, such as autonomous driving, intelligent surveillance, and IoT devices.

Developer Recommendations

  • Model Adaptation: Developers can try to adapt the sLoTh framework to other types of Transformer architectures to explore its application potential in different fields.
  • Performance Optimization: Further optimize the seLoRA and shared neuronal threshold modulation mechanisms to improve the model's adaptation speed and accuracy.
  • Application Scenario Expansion: Explore the application of sLoTh in multi-modal learning, cross-task migration, and other fields to expand its scope of application.

Source: ArXiv cs.CV (2026-08-27)

— END —

Tags: #Sparse Transformer #Continual Learning #Edge AI #Low-Rank Adaptation #Parameter Efficiency

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…