LR-Merge Predicts and Corrects Model-Merging Interference with Receiver-Local Derivatives
Summary:The paper shifts model-merging interference estimation from post-merge evaluation to a first-order directional derivative evaluated at the receiver model. The derivative predicts nearby merges per task and per example; in the reported vision experiments, it halves the accuracy error relative to a no-change baseline and identifies most prediction flips. LR-Merge freezes receiver-local responses to form an affine surrogate over merge coefficients, then optimizes those coefficients with convex tea
Core Result
The main cost of model merging is rarely the arithmetic of adding task vectors. It is the need to construct and evaluate a merged model for every candidate coefficient. This paper moves that cost upstream: at the receiver model θ_R, it measures the local sensitivity of task A's outputs to task B's task vector, then uses that sensitivity to predict interference from nearby merges. The abstract reports that, in vision experiments, the local derivative halves the accuracy error relative to a no-change baseline and identifies most examples whose predictions flip.
This is not an approximation that can be applied at any point in weight space. The paper provides an important counterexample: when the same derivative is evaluated at the shared pretrained backbone, it is nearly orthogonal to realized changes in the vision experiments and only partly predictive in the LoRA language-model pilots. Interference therefore depends on the local parameter geometry of the receiving model rather than on parameter distance alone.
How the First-Order Derivative Predicts Interference
Let the shared pretrained backbone be θ_0, and let the task vector for task i be τ_i = θ_i - θ_0. A merged model can be written as:
θ_merge(λ) = θ_0 + Σ_i λ_iτ_i
For the output function f_A of task A, the local sensitivity used in the work can be summarized as:
D_A←B^R(x) = [d/dε f_A(x; θ_R + ετ_B)]_(ε=0) = J_A(θ_R)τ_B
Here, J_A(θ_R) is the Jacobian of task A's outputs with respect to model parameters, while J_Aτ_B is the directional derivative along task B's vector. It answers a narrower but more useful question than a global similarity score: if task B's parameter direction is added near the current receiver model, in which direction and by how much will each task-A output change immediately?
The prediction relies on two conditions. First, it is most applicable to merges near the receiver model. Continued fine-tuning may alter the local tangent space and task couplings. Second, one linearization is insufficient for the complete multi-task merge path. The paper consequently requires two evaluations along that path rather than relying on a single anchor point. Controls attribute the failure observed at the pretrained backbone to displacement along the receiver's own task-vector direction, not simply to parameter distance; this mismatch appears within the first few dozen fine-tuning steps.
From an implementation perspective, explicitly materializing an output-by-parameter Jacobian with O outputs and P parameters requires O×P storage and is generally impractical. JVP, Jacobian-vector products, or finite-difference approximations may be more feasible, but the abstract does not specify the method used, its estimation error, or its memory overhead. These remain important reproduction gaps.
LR-Merge: Turning Merge Search into Convex Matching
LR-Merge freezes receiver-local responses to construct an affine surrogate over the merge coefficients. A generic representation is:
ŷ_i(x; λ) ≈ y_R(x) + G_iλ
Here, y_R(x) is the receiver model's output on a probe, and the columns of G_i correspond to the receiver-local directional responses for different task vectors. Outputs from the task-specific teacher models serve as matching targets t_i(x). The coefficients can then be optimized through teacher matching over unlabeled probes:
min_λ Σ_i w_i ||ŷ_i(x; λ) - t_i(x)||²
Once the surrogate matrix is fixed, this can be organized as a convex matching problem over the merge coefficients. Instead of running the complete merged model for every candidate coefficient, LR-Merge solves for the coefficients on the surrogate and performs exact evaluation only on selected candidates.
“Label-free” does not mean “supervision-free.” The method still needs input probes and outputs from each task teacher. Any bias in those teachers can be inherited by the surrogate. If the probe distribution does not cover production traffic, the surrogate may recommend unsafe coefficients outside that distribution. Probe sampling and the relative weighting w_i used to balance conflicting tasks are therefore likely to be important engineering choices.
The abstract states that no gradient needs to pass through the merged network. That does not establish zero-gradient or zero-cost construction: computing receiver-local derivatives may still require backpropagation through the receiver model. The abstract does not disclose the cost of derivative computation or surrogate assembly, so those overheads should not be inferred from the optimization stage alone.
Engineering Trade-offs
- One-time setup versus repeated queries: Constructing local responses and the surrogate has an upfront cost, but a completed surrogate answers new coefficient and task-subset queries in seconds. The method is most attractive when coefficients, task combinations, or merge paths must be searched repeatedly, not for a one-off merge.
- Locality creates drift risk: A first-order surrogate is most reliable for nearby merges. If the receiver continues training, task vectors are regenerated, or the merge moves much farther, the surrogate may become stale and must be rebuilt.
- One derivative is not enough for general multi-task merging: Local linearity may be adequate for two tasks, but additional tasks introduce cross-task coupling and path nonlinearity. The paper therefore uses two evaluations along the merge path.
- The gain is not an order-of-magnitude quality advantage: The paper reports that exact teacher matching through the merged network takes roughly three times longer to converge but performs similarly at an equal compute budget. LR-Merge's main benefit is therefore lower interaction latency and better amortization, not necessarily higher final quality under identical compute.
- Architecture transfer cannot be assumed: LR-Merge improves the tested closed-form, isotropic, and sparsified base mergers, while responses from the pretrained backbone can degrade the base merge. The LoRA pilots also show weaker predictive power than the vision experiments. Full-model and adapter merging need separate calibration.
Evaluation Results and Caveats
The abstract reports that the receiver-local derivative predicts nearby merges per task and per example; that it halves accuracy error relative to the no-change baseline; that it identifies most prediction-flipping examples; and that LR-Merge improves every tested base merger, with gains generally increasing as more tasks are merged. Once constructed, the surrogate reportedly answers new coefficient and task-subset queries in seconds.
The evidence does not yet establish universal superiority over existing merge-search methods. “Halving accuracy error” is a relative result, not a doubling of the merged model's absolute accuracy. “Identifying most flips” is reported without precision, recall, or confidence intervals. The abstract also omits the models, datasets, task counts, named base mergers, absolute scores, random seeds, statistical variance, and a precise definition of the no-change baseline. “All tested mergers” may cover only a limited experimental scope and should not be extrapolated to other architectures. The weaker LoRA results also make the language-model evidence less robust than the vision evidence.
The work is currently best viewed as a merge-search accelerator with a clear mechanistic basis, not as an established universal optimum. Reproduction should prioritize the derivative estimator, surrogate-construction cost, cross-task coefficient separation, and degradation after the receiver model continues training.
Developer and Deployment Guidance
- Target repeated-search workloads: Use LR-Merge when coefficients, task subsets, or sparse masks must be evaluated many times. For a single merge, direct exact evaluation remains simpler.
- Build a representative unlabeled probe set: Stratify probes by task, input length, difficulty, and production-traffic distribution. Keep a separate validation probe set that is excluded from optimization to detect distributional overfitting.
- Use two path anchors for multi-task merges: Build the surrogate at two points along the merge path and compare it with a single-point ablation, rather than assuming multi-task interference is linear.
- Validate candidates exactly before deployment: Run the actual merged model on selected candidates returned by the surrogate and check target-task accuracy, forgetting, and prediction flips. The surrogate should accelerate search, not replace acceptance testing.
- Monitor fine-tuning drift: Record receiver fine-tuning steps, task-vector generation time, and the underlying model version. Rebuild the local responses after continued training or replacement of task heads.
- Evaluate full-parameter and LoRA merging separately: LoRA's lower-dimensional parameter space does not guarantee a more reliable derivative. Measure prediction error and surrogate gains separately for full-parameter models, LoRA, closed-form, isotropic, and sparsified mergers.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-21)
Tags: #Model Merging #Task Vectors #Directional Derivatives #Label-Free Optimization #LoRA
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments