arXiv Introduces Jacobian Lens Pruning: Enhancing Depth Pruning for Large Language Models

Summary:arXiv introduces the Jacobian Lens (J-Lens) neural artifact to enhance depth pruning for large language models (LLMs). By mapping perturbations at a layer to the final residual stream, J-Lens addresses the limitation of traditional Block Influence (BI) methods that only focus on minimizing hidden state rotation. Experiments on models like Qwen3-8B, Llama-3.1-8B-IT, and Gemma-3-12B-IT demonstrate that J-Lens pruning improves post-pruning accuracy by up to 19 percentage points at a common sparsity

---

Technical Mechanism Analysis

The Jacobian Lens (J-Lens) is a novel neural network tool designed to address the limitations of traditional pruning methods that focus solely on local redundancy while ignoring downstream impacts. Its core mechanism involves mapping perturbations at each layer to the final residual stream, allowing for a more accurate assessment of each Transformer's impact on the model's output. Specifically, J-Lens operates through the following steps:

1. Perturbation Mapping: Each Transformer's perturbations are mapped to the final residual stream, generating an influence graph.
2. Impact Scoring: Based on the influence graph, importance scores for each block are calculated, prioritizing the pruning of blocks with less impact on the model's output.
3. Pruning Decision: At a common sparsity level, the J-Lens pruning method selects blocks to prune based on these scores.

Engineering Trade-offs and Experimental Performance

Experimental results demonstrate that the J-Lens pruning method performs well across multiple models:

• Accuracy Improvement: The J-Lens pruning method improves post-pruning accuracy by up to 19 percentage points on models like Qwen3-8B, Llama-3.1-8B-IT, and Gemma-3-12B-IT.
• Perplexity Reduction: Compared to traditional BI methods, J-Lens pruning significantly reduces perplexity damage.

However, the J-Lens pruning method also faces some challenges:

• Computational Cost: Due to the need to calculate perturbation mappings for each block, the computational cost of J-Lens pruning is high, which may not be suitable for real-time applications.
• Model Dependency: The effectiveness of J-Lens pruning may vary depending on the model architecture and task, requiring optimization for specific applications.

Developer Implementation and Deployment Recommendations

For developers looking to adopt the J-Lens pruning method, here are some recommendations:

1. Model Selection: Prioritize models with good initial performance to fully leverage the advantages of J-Lens pruning.
2. Compute Resources: Ensure sufficient compute resources to support the computational demands of J-Lens pruning, especially when dealing with large models.
3. Optimization Strategies: Combine other pruning and quantization techniques to further optimize model performance.
4. Experimental Validation: Conduct thorough experimental validation in specific application scenarios to determine the best pruning strategy.

Conclusion

The J-Lens pruning method offers a new approach to AI model compression and optimization, particularly excelling in handling large language models. Although it has high computational costs, with reasonable optimization strategies, it can achieve efficient resource utilization and performance improvements in large-scale AI applications.

---

Source: ArXiv Machine Learning (cs.LG) (2026-11-03)

— END —

Tags: #Depth Pruning Follow #Jacobian Lens Follow #Large Language Models Follow #Model Compression Follow