ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #Multi-Task Learning #Model Merging #Sparse Representation #Qwen Model #Deep Learning

ArXiv Proposes Sparse-Representation-Based Model Merging Framework to Address Multi-Task Interference

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:The ArXiv team proposes a novel sparse-representation-based model merging framework that leverages Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. This approach addresses the task interference problem in multi-task model merging. Coupled with a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) for selective layer merging, experiments on Qwen2.5-1.5B and Qwen2.5-7B demonstrate superior pe


Background and Challenge

Multi-task model merging is an efficient approach to building multi-task generalist models without additional training. However, its performance often degrades under severe task interference. Task interference primarily stems from the entanglement of task-specific features within the parameter space, making traditional decomposition methods insufficient for effectively isolating useful task directions from interfering components.

Method and Innovation

The ArXiv team proposes a novel sparse-representation-based model merging framework with the following key innovations:

  1. Sparse Autoencoders (SAEs): Project task vectors into a high-dimensional sparse feature space to achieve feature-level disentanglement.
  2. Lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO): Identifies task-critical layers for selective merging, reducing computational overhead.

Experiments and Results

Experiments on Qwen2.5-1.5B and Qwen2.5-7B models demonstrate the framework's superior performance across the following tasks:

  • Mathematical Reasoning: Excels in solving complex mathematical problems.
  • Code Generation: Improves the accuracy and efficiency of generated code.
  • Instruction Following: Enhances the ability to understand and execute complex instructions.
  • Common Sense Tasks: Performs more reliably in handling common sense questions.

Notably, in a highly conflicting four-task setting, the method achieves a 2.78% improvement over the strongest baseline.

Industry Impact and Developer Recommendations

This research offers a new technical path for multi-task model merging, particularly in scenarios requiring the handling of complex task combinations. Developers can consider the following recommendations:

  • Task Disentanglement: Incorporate sparse representation methods for task disentanglement in multi-task model design.
  • Selective Merging: Use optimizers like GR-ZOO for selective layer merging to balance performance and computational cost.
  • Experimental Validation: Conduct thorough experiments across different task combinations and model scales to validate the method's effectiveness.

Source: ArXiv cs.AI (2026-08-26)

— END —

Tags: #Multi-Task Learning #Model Merging #Sparse Representation #Qwen Model #Deep Learning

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…