FusionEval: A New Benchmark for Evaluating Model Fusion in Large Language Models

Summary:ArXiv introduces FusionEval, a benchmark designed to evaluate the effectiveness of model fusion techniques for large language models (LLMs). FusionEval provides controlled expert pools, tests fusion methods with and without additional training, and offers a unified evaluation suite for reasoning, tool use, and computational cost. Experiments show that training-free methods preserve reasoning performance but degrade tool use significantly, while MOPD preserves tool use but requires additional dat

---

Core Breakthrough

ArXiv has released FusionEval, a benchmark tool designed to evaluate the effectiveness of model fusion techniques for large language models (LLMs). The key innovations of FusionEval include:

1. Controlled Expert Pools: FusionEval provides expert pools independently trained from Qwen3 and Llama base models across domains such as math, code, science, instruction following, and tool use.

2. Comprehensive Evaluation Framework: The benchmark not only focuses on average performance but also emphasizes the evaluation of long-chain reasoning and tool use to avoid masking individual expert capability losses.

3. Fusion Method Testing: FusionEval tests fusion methods with and without additional training and reports normalized average and worst performance, runtime, and peak GPU memory usage.

Technical Mechanism Analysis

FusionEval achieves the in-depth evaluation of model fusion through the following approaches:

• Expert Pool Construction: The expert pools are built by independently training domain experts from Qwen3 and Llama base models, ensuring the independence and diversity of expert capabilities.

• Fusion Method Integration: FusionEval integrates various fusion methods, including training-free methods and methods requiring additional training (such as MOPD), and compares them through a unified evaluation framework.

• Multi-Dimensional Evaluation: The evaluation metrics include reasoning performance, tool use capability, runtime, and GPU memory usage, providing a comprehensive reflection of the actual effects of model fusion.

Engineering Trade-offs and Experimental Performance

Experimental results show that:

• Training-free Methods: Perform well in reasoning but degrade significantly in tool use. This indicates the limitations of training-free methods in preserving expert capabilities.

• MOPD Method: Performs better in tool use but requires additional data preparation and training costs. This shows that the MOPD method has advantages in preserving tool use capabilities but requires higher computational resources.

Developer Implementation and Deployment Recommendations

FusionEval provides the following recommendations for developers:

• Choose the Right Fusion Method: Select the appropriate fusion method based on the specific application scenario and resource constraints. For example, in resource-constrained environments, training-free methods can be prioritized, while for high-precision tool use, methods like MOPD that require additional training should be chosen.

• Focus on Long-Chain Reasoning and Tool Use: When evaluating the effectiveness of model fusion, special attention should be paid to the performance of long-chain reasoning and tool use, not just average performance.

• Optimize Computational Costs: The evaluation framework of FusionEval can help developers better understand the computational costs of different fusion methods for optimization.

Conclusion

FusionEval provides important support for future research on efficient model fusion. Its comprehensive evaluation framework and controlled expert pools offer valuable references for developers.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-24)

— END —

Tags: #FusionEval Follow #Model Fusion Follow #Benchmark Follow #Large Language Models Follow #Computational Cost Follow