ArXiv Proposes Sparse-Representation-Based Model Merging Framework to Address Multi-Task Interference
Summary:The ArXiv team proposes a novel sparse-representation-based model merging framework that leverages Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. This approach addresses the task interference problem in multi-task model merging. Coupled with a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) for selective layer merging, experiments on Qwen2.5-1.5B and Qwen2.5-7B demonstrate superior pe
Background and Challenge
Multi-task model merging is an efficient approach to building multi-task generalist models without additional training. However, its performance often degrades under severe task interference. Task interference primarily stems from the entanglement of task-specific features within the parameter space, making traditional decomposition methods insufficient for effectively isolating useful task directions from interfering components.
Method and Innovation
The ArXiv team proposes a novel sparse-representation-based model merging framework with the following key innovations:
- Sparse Autoencoders (SAEs): Project task vectors into a high-dimensional sparse feature space to achieve feature-level disentanglement.
- Lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO): Identifies task-critical layers for selective merging, reducing computational overhead.
Experiments and Results
Experiments on Qwen2.5-1.5B and Qwen2.5-7B models demonstrate the framework's superior performance across the following tasks:
- Mathematical Reasoning: Excels in solving complex mathematical problems.
- Code Generation: Improves the accuracy and efficiency of generated code.
- Instruction Following: Enhances the ability to understand and execute complex instructions.
- Common Sense Tasks: Performs more reliably in handling common sense questions.
Notably, in a highly conflicting four-task setting, the method achieves a 2.78% improvement over the strongest baseline.
Industry Impact and Developer Recommendations
This research offers a new technical path for multi-task model merging, particularly in scenarios requiring the handling of complex task combinations. Developers can consider the following recommendations:
- Task Disentanglement: Incorporate sparse representation methods for task disentanglement in multi-task model design.
- Selective Merging: Use optimizers like GR-ZOO for selective layer merging to balance performance and computational cost.
- Experimental Validation: Conduct thorough experiments across different task combinations and model scales to validate the method's effectiveness.
— END —Source: ArXiv cs.AI (2026-08-26)
Tags: #Multi-Task Learning #Model Merging #Sparse Representation #Qwen Model #Deep Learning
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments