Deletion Tuning Matches Full Fine-Tuning by Zeroing Under 0.05% of Weights
Summary:Deletion Tuning learns a binary mask over the gate projection of Transformer blocks, replacing selected pretrained weights with zeros while leaving all other parameters untouched. The paper reports that changing fewer than 0.05% of weights can match full fine-tuning across reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. The method also produces checkpoints that are hundreds to thousands of times smaller than full fine-tuning checkpoints and ex
Core Idea: Reframing Post-Training as Weight Deletion
Deletion Tuning learns a binary mask over the gate projection of Transformer blocks. Selected weights are set to zero, while all remaining parameters stay at their pretrained values. The resulting model is therefore a structurally pruned model rather than a full-parameter update or a LoRA-style low-rank adapter.
The paper reports that zeroing fewer than 0.05% of weights can match full fine-tuning at scale in three post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. This does not mean that a tiny parameter set stores all model knowledge. Rather, some post-training objectives may be achieved by removing selected connections and changing how existing capabilities are routed through the network.
Technical Mechanism
For a gate projection matrix
[ W \in \mathbb{R}^{d_{out}\times d_{in}}, ]
the method learns (M\in{0,1}^{d_{out}\times d_{in}}) and produces:
[ \widetilde{W}=M\odot W. ]
A mask entry equal to zero deletes the corresponding weight, while a one preserves the original value. After the mask is merged, inference follows the original computation graph, with no adapter branch or dynamic routing required.
Compared with LoRA, which typically stores two low-rank matrices representing (W+BA), Deletion Tuning only needs a binary mask. Under bit-packed or compressed storage, this can produce checkpoints hundreds or thousands of times smaller than full fine-tuning. The practical compression ratio, however, depends on mask encoding and model-loading overhead.
Why Sparse Deletion Can Approximate Full Fine-Tuning
The paper analyzes parameter geometry after reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. On all three measures, Deletion Tuning is closer to full fine-tuning than LoRA or random sparse fine-tuning.
A plausible interpretation is that many post-training updates do not require dense continuous movement across an entire weight matrix. They may instead exploit a small set of structurally effective changes. Deleting fixed pretrained weights cannot synthesize every possible new direction, but it can alter gate paths and recombine capabilities already represented in the network.
Geometric similarity does not imply functional equivalence. Stable rank and spectral drift are descriptive, and they do not establish that the two methods form the same internal representation or generalize under distribution shift. The critical follow-up questions include whether the mask transfers across tasks, whether selected layers are necessary, and whether the method works beyond a narrow family of models and training data.
Engineering Trade-offs
The clearest advantages are smaller distributed artifacts and reduced catastrophic forgetting. In reinforcement learning, prior-task accuracy remains close to the base-model level, whereas full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points.
Training cost is less certain. If the binary mask is learned by iteratively zeroing weights, restoring candidates, and evaluating task performance, optimization may require many forward passes. A small final mask does not automatically imply cheap post-training. The supplied summary does not disclose mask-search cost, candidate evaluations, or GPU hours.
Lower forgetting may also reflect weaker adaptation rather than better continual learning. Evaluation must therefore compare new-task performance, prior-task retention, general capability regression, and total compute under matched budgets.
What the Evaluation Must Control
The reported claims need scrutiny around model scale, task coverage, random seeds, statistical significance, and compute matching. In particular:
- Full fine-tuning, LoRA, and sparse tuning should receive comparable training-token and GPU-time budgets.
- The three post-training regimes should be tested across multiple model families and task domains.
- Restricting deletion to gate projections should be compared with other projection types and layer selections.
- Masks should be evaluated under task and distribution shifts.
- Forgetting assessment should extend beyond task accuracy to instruction following, safety behavior, and general capability.
Deployment Guidance
Teams that frequently distribute model adapters or need to preserve base-model behavior should treat Deletion Tuning as an alternative to LoRA and full fine-tuning. Reproductions should track mask density, selected layers, prior-task regression, artifact size, training wall-clock time, and end-to-end serving latency.
The mask can be merged directly into the weights or stored as a compressed bitmap and applied during loading. If the serving stack supports sparse kernels, it is worth checking whether it accelerates computation; otherwise, the method may reduce artifact size without reducing inference FLOPs.
Deletion Tuning's most important contribution is conceptual: parameter-efficient post-training need not always create a small continuous update. It can instead select a compact set of reversible structural deletions. If its transferability is independently confirmed, this approach could become useful for continual learning and lightweight model distribution; if mask search is computationally expensive, its benefits will be concentrated more on deployment than on training.
— END —Source: Hugging Face Daily Papers (2026-11-07)
Tags: #Deletion Tuning #Parameter-Efficient Fine-Tuning #Binary Masking #Continual Learning #LLM Post-Training
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments