Oriel — AI & technology worth noticing, every day.

Emergent Low-Dimensional Topological Dynamics Dictate Generalization and Escape Mechanics in Billion-Parameter Neural Networks

Paper · https://arxiv.org/abs/2611.08502

arXiv:2611.08502v1 Announce Type: new Abstract: The remarkable generalization of modern over-parameterized neural networks defies classical statistical learning bounds. Here we establish that optimization trajectories in billion-parameter models do not sample the ambient dimension $\mathbb{R}^N$ homogeneously; instead, they are dynamically constrained to a low-dimensional execution submanifold $\mathcal{M}_{\text{exec}}$ (with empirically observed effective dimension $d_{\text{eff}} \le 12$). By equipping the parameter space with a regularized Hessian metric $g_{ij} = \nabla_i \nabla_j L + \epsilon \delta_{ij}$ and analyzing stochastic Langevin diffusion on loss sublevel sets, we derive a rigorous Topological Generalization Bound grounded in projected subspace persistent homology. Furthermore, by linking Morse index transitions to Betti number annihilation ($\beta_1 \to 0$), we formulate an exact mathematical mechanism for exponential escape from strict saddles under anisotropic noise. We validate our framework across 80+ architectures (ResNets, Vision Transformers, and 7B LLMs). To bridge theoretical topology with computational feasibility, we resolve the curse of dimensionality by computing projected persistent homology proxies via randomized Lanczos iterations restricted strictly within active Hessian subspaces. Building upon this mathematical foundation, we introduce Topological-Adaptive Optimization (Top-SGD and Top-AdamW)---a family of wrapper algorithms that dynamically modulate learning rates using active curvature signatures. Extensive benchmarks demonstrate a 14.1\% reduction in total wall-clock training time on ImageNet with all computational and memory overheads fully accounted for. Code is publicly available at https://github.com/imslab-research/Topological-SGD.

First seen by Oriel on .

AI summary

This paper proposes that billion-parameter neural-network optimization follows low-dimensional topological dynamics and introduces curvature-adaptive optimizers that reportedly reduce ImageNet training time by 14.1%.

  • The authors report an execution submanifold with empirically observed effective dimension d_eff ≤ 12 across 80+ architectures, including ResNets, Vision Transformers, and 7B LLMs.
  • They combine a regularized Hessian metric, stochastic Langevin diffusion, and projected persistent homology to derive a Topological Generalization Bound.
  • The proposed mechanism links Morse index transitions to Betti number annihilation (β_1 → 0), explaining exponential escape from strict saddles under anisotropic noise.
  • Top-SGD and Top-AdamW dynamically adjust learning rates using active curvature signatures while restricting persistent-homology computation to active Hessian subspaces.
  • Benchmarks report a 14.1% reduction in total ImageNet training time after accounting for computational and memory overheads; code is publicly available.

#neural networks #persistent homology #optimization #topological dynamics

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.08502v1https://arxiv.org/abs/2611.08502 Read discussion