Deletion Tuning Matches Full Fine-Tuning by Zeroing Just 0.05% of LLM Weights

Summary:Deletion Tuning learns a binary mask over the gate projections of Transformer blocks and zeroes fewer than 0.05% of pretrained weights without updating the remaining parameters. The paper reports that this sparse approach matches full fine-tuning across reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Its masks are hundreds to thousands of times smaller than conventional checkpoints, while continual-learning experiments show substantially less c

---

Core Finding: Post-Training Does Not Necessarily Require Updating Every Weight

Deletion Tuning reframes post-training as a sparse decision problem. It freezes the pretrained model, learns a binary mask over the gate projections of Transformer blocks, and zeroes the selected weights without updating the remaining parameters. The paper reports that changing fewer than 0.05% of pretrained weights is sufficient to match full fine-tuning across three regimes:

• Reinforcement learning with verifiable rewards (RLVR);
• Supervised fine-tuning (SFT);
• On-policy distillation.

Unlike LoRA, the method does not represent the adaptation as a low-rank product. Its central hypothesis is that the effective adjustment required for a new task may be concentrated in a small set of gate-projection parameters. Disabling selected paths can therefore acquire the new behavior while preserving more of the model's prior-task functionality than a dense update.

Technical Mechanism: A Binary Mask over Gate Projections

For a gate-projection matrix (W), the method can be expressed as:

[ W_{\theta}^{\prime} = W_0 \odot (1-M), \qquad M_{ij}\in{0,1} ]

Here, (W_0) is the frozen pretrained matrix, (M) is the trainable binary mask, and (W_{\theta}^{\prime}) is the deployed weight matrix. The optimizer does not directly update continuous weight values; it determines which gate-projection entries should be zeroed.

Gate projections sit on conditional-computation paths in a Transformer and regulate how residual or expert branches influence hidden states. Selecting them is therefore not arbitrary sparsification. The method changes which existing pathways are active, treating post-training as a learned structural intervention.

The binary mask may be optimized through continuous relaxation, thresholding, or another differentiable approximation. At deployment, however, only the selected zero positions need to be stored. A bit mask or compact index representation can be hundreds to thousands of times smaller than a full fine-tuning checkpoint.

Why Masking Can Approximate Dense Fine-Tuning

The paper investigates update geometry rather than attributing the result merely to sparsity regularization. It compares Deletion Tuning with LoRA and random sparse fine-tuning using stable rank, spectrum drift, and principal-weight overlap.

Stable Rank

A common expression for stable rank is:

[ \operatorname{sr}(W)=\frac{|W|_F^2}{|W|_2^2} ]

It measures the effective number of dominant directions in a matrix. Full fine-tuning can produce a richer rearrangement of the weight spectrum. Deletion Tuning is more constrained, yet the paper reports that its stable-rank behavior is closer to FFT than that of LoRA or random sparse fine-tuning. This suggests that low rank is not necessarily the best way to approximate dense adaptation; parameter location and update structure may matter more than parameter count.

Spectrum Drift

Spectrum drift captures how training redistributes matrix energy across singular-vector directions. A sparse deletion should alter fewer directions than a dense update, but it must still concentrate changes in meaningful locations. The reported proximity to FFT indicates that the learned mask is not simply removing arbitrary weights.

Principal-Weight Overlap

Principal-weight overlap compares the dominant update directions produced by different methods. Deletion Tuning is reported to align more closely with FFT than the sparse baselines, providing a geometric explanation for its stronger similarity to dense fine-tuning.

Engineering Trade-Offs: Small Checkpoints Do Not Imply Cheaper Training

The clearest advantage of Deletion Tuning is deployment and storage, not necessarily training throughput.

Storage and Distribution

Full fine-tuning generally requires a complete model copy for each task. LoRA stores a low-rank delta, while Deletion Tuning stores only the selected zero positions in gate projections. This can substantially reduce artifact size for frequently updated model fleets, edge deployments that maintain multiple task versions, and low-bandwidth model-distribution systems.

Training Cost

Binary parameters are not automatically cheaper to optimize. Learning the mask can still require substantial forward and backward computation. If it is implemented through elementwise gates, thresholding, or combinatorial search, memory use, runtime, and hyperparameter sensitivity may be comparable to or even worse than LoRA. The abstract does not fully disclose the mask optimizer, number of training steps, memory footprint, or throughput, so a smaller final checkpoint should not be interpreted as faster training.

Expressive Capacity

A mask can delete weights but cannot directly introduce new numerical values as full fine-tuning can. The method works best when the target behavior can be obtained by turning off selected existing pathways. Tasks that require substantial new knowledge, output conventions, or fundamentally new computation may favor updates that can add and alter parameters.

Continual Learning: Less Forgetting of Prior Capabilities

One of the paper's most consequential results appears in the RLVR setting. Prior-task accuracy under Deletion Tuning remains near the base-model level, whereas FFT, LoRA, and random sparse fine-tuning lose as much as 5.4 percentage points.

This does not mean catastrophic forgetting has been eliminated. Freezing the main weights does not guarantee that old abilities remain untouched, because deleting a gating branch can alter computation used by other tasks. The stronger conclusion is that, under the reported experiments, the constrained update causes less degradation than the comparison methods.

For continual post-training, the method also offers a useful deployment model: task versions can be represented by small masks instead of long-lived dense checkpoints. Whether masks can be safely composed, and whether tasks that remove the same paths interfere with one another, still requires dedicated evaluation.

Benchmark Credibility and Open Questions

Covering three post-training regimes is more informative than reporting a single SFT benchmark. However, the claim that Deletion Tuning matches FFT still requires detailed verification:

• Scale and architecture coverage: Does the conclusion hold across model sizes, context lengths, and Dense and MoE architectures?
• Training-budget parity: Do FFT, LoRA, random sparse fine-tuning, and Deletion Tuning use identical data, optimization steps, and compute?
• Statistical significance: Are gains and forgetting reductions robust across repeated runs?
• Baseline completeness: Are stronger parameter-efficient methods included, such as sparse experts, mask-based tuning, and modern low-rank adapters?
• Inference cost: If masked weights remain in dense matrices, storage improves while compute does not. The distinction between sparse storage and sparse execution is essential.

The claim also depends on the relationship between the new task and the pretrained model's existing capabilities. If the target lies far outside the base model's competence distribution, binary deletion may not provide enough functional reorganization.

Practical Deployment Guidance

Deletion Tuning is best positioned as a compact representation for task-specific model versions and a candidate for low-forgetting continual post-training, rather than an automatic replacement for every fine-tuning method.

Engineering teams should:

1. Establish task success, prior-task accuracy, training memory, and inference-latency baselines for FFT and LoRA.
2. Compare binary-mask optimizers and relaxation strategies under an equal training budget.
3. Measure artifact savings while distinguishing sparse storage from sparse computation.
4. Test mask composition, task switching, and interference across multiple continual-learning stages.
5. Examine scaling, MoE architectures, and knowledge-intensive tasks to identify the method's limits.

If the objective is to reduce model-distribution size, manage frequent task versions, or suppress forgetting, the method deserves a place in the experimental pipeline. If the priority is minimum training cost or rapid injection of entirely new knowledge, FFT, LoRA, and other parameter-efficient methods should remain active competitors.

---

Source: Reddit r/LocalLLaMA (2026-11-10)

— END —

Tags: #Deletion Tuning Follow #LLM Post-Training Follow #Binary Masking Follow #Parameter-Efficient Fine-Tuning Follow #Continual Learning Follow