Oriel — AI & technology worth noticing, every day.

Adaptive Depth Aggregation in Graph Transformers for Long-Range Graph Reasoning

Paper · https://arxiv.org/abs/2611.08500

arXiv:2611.08500v1 Announce Type: new Abstract: Graph Transformers improve long-range node-to-node communication through global or structure-aware interactions. However, improving cross-node communication does not determine how computations produced at different depths should be combined. Standard residual streams accumulate successive computations with fixed unit coefficients shared across nodes. This fixed aggregation is a residual-interface constraint: once intermediate contributions have been produced, the interface cannot explicitly reweight them by node. We hypothesize that this constraint can create a depth-wise aggregation mismatch when the relative utility of these intermediate contributions varies across nodes and structural contexts. To study this hypothesis, we formalize Graph Transformer residual computation as a causal node-wise history of graph-propagation and feed-forward network (FFN) contributions, making depth aggregation an explicit interface choice. We realize node-adaptive aggregation over this history with Attention Residuals (AttnRes) while preserving the backbone's graph-propagation and auxiliary-state machinery. Block AttnRes reduces the routing cost of long histories relative to Full AttnRes by pooling consecutive contributions. We instantiate the interface in GPS and GRIT and evaluate it on all five ECHO tasks. Both variants achieve lower mean absolute error (MAE) than their independently tuned backbones on every task, and at least one obtains the lowest mean among all compared methods in each task. Routing-only donor-swap interventions support the mismatch hypothesis, while additional ablations examine the key design choices. Together, these results support node-adaptive depth aggregation as a useful complement to cross-node communication in long-range graph reasoning.

First seen by Oriel on .

AI summary

This paper introduces node-adaptive Attention Residuals for aggregating Graph Transformer computations across depth, improving long-range graph reasoning over tuned GPS and GRIT backbones.

  • Standard residual streams combine depth-wise contributions with fixed unit coefficients, preventing node-specific reweighting of intermediate computations.
  • Attention Residuals treat graph-propagation and FFN outputs as a causal node-wise history and learn adaptive aggregation over that history.
  • Block AttnRes pools consecutive contributions to reduce the routing cost of long histories compared with Full AttnRes.
  • Across all five ECHO tasks, both variants achieved lower MAE than their independently tuned backbones, with one variant attaining the lowest mean on each task.
  • Routing-only donor-swap interventions support the proposed depth-aggregation mismatch hypothesis, while ablations assess key design choices.

#graph transformers #attention residuals #long-range reasoning #graph neural networks

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.08500v1https://arxiv.org/abs/2611.08500 Read discussion