Output-Aware Block Influence: Jacobian-Lens Weighting for Depth Pruning
Paper · https://arxiv.org/abs/2611.00015
arXiv:2611.00015v1 Announce Type: new Abstract: Depth pruning removes entire transformer blocks and therefore depends on a reliable criterion for identifying which blocks can be removed. Block Influence (BI) ranks blocks by how minimally they rotate the hidden state. However, local redundancy does not necessarily imply downstream irrelevance. We address this limitation using the Jacobian Lens (J-Lens) neural artifact, a map transporting perturbations at a layer to the final residual stream. At a common sparsity level, pruning scores applying J-Lens transportation to $BI$, what we call $J-BI$, were found to improve post-pruning accuracy by up to 19 points over $BI$ while reducing damage to perplexity for Qwen3-8B, Llama-3.1-8B-IT, and Gemma-3-12B-IT. We also perform a single-block ablation study and use J-Lens probing to explain why the proposed scores improved performance.
First seen by Oriel on .
AI summary
The paper proposes J-BI, a depth-pruning score that uses Jacobian Lens to account for how block perturbations affect the final residual stream, improving pruning outcomes over Block Influence.
- Block Influence ranks transformer blocks by how minimally they rotate the hidden state, but local redundancy may not mean a block is irrelevant downstream.
- J-Lens transports layer-level perturbations to the final residual stream, and J-BI applies this transport to Block Influence scores.
- At a common sparsity level, J-BI improved post-pruning accuracy by up to 19 points over BI and reduced perplexity damage.
- The evaluation covers Qwen3-8B, Llama-3.1-8B-IT, and Gemma-3-12B-IT, with single-block ablations and J-Lens probing.
#transformers #model pruning #jacobian lens
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.00015v1 | https://arxiv.org/abs/2611.00015 Read discussion |