Path-Conditioned Attention Mobility: From Depth to Execution Hubs in SegFormer

Paper · https://arxiv.org/abs/2610.23893

arXiv:2610.23893v1 Announce Type: new Abstract: Transformer compression often couples two distinct quantities: retained attention capacity and the number of sequential depths at which that capacity is executed. We study whether useful attention functions in a hierarchical vision transformer are locally mobile across depth. Using SegFormer-B5 for human segmentation, we find that some task-relevant attention heads can be relocated to nearby depths, but only when their computational context is respected. Head-only relocation is insufficient, while joint adaptation of the relocated head and surrounding representation boundary improves functional preservation; replacing donor-specific normalization and spatial-reduction preprocessing causes large degradation. We use this path-conditioned mobility to reorganize retained attention from depth into width. Starting from a fixed parent with 120 retained Stage-3 heads across 40 attention execution depths, we consolidate the same 120-head budget into 20 execution hubs. Under a layout-matched three-seed recovery control, K20 reaches 0.960686 +/- 0.000059 mIoU versus 0.960400 +/- 0.000074 for the corresponding State120 control. The paired point difference is +0.000285, while a paired image-bootstrap confidence interval includes zero; we therefore interpret the result as no measurable quality penalty rather than statistical superiority. For deployment, FOLDED_VEC preserves donor-specific paths while vectorizing their execution. Relative to the same 120-head State120 parent, it reduces end-to-end NVIDIA L4 latency by 5.28%, 3.15%, and 3.58% at batch sizes 1, 4, and 8. These results suggest that attention redundancy includes not only removable capacity but also execution-depth redundancy, provided that the computational path defining a relocated function is preserved.

First seen by Oriel on Oct 20, 2026, 05:50 UTC.