Oriel — AI & technology worth noticing, every day.

Classification with Large-Step Gradient Descent: Unifying Self-Stabilization and Convergence Rates

Paper · https://arxiv.org/abs/2611.17459

arXiv:2611.17459v1 Announce Type: new Abstract: Neural networks are often trained with gradient descent (GD) using step sizes that exceed classical stability thresholds, a regime where standard convergence theory no longer applies. Despite exhibiting non-monotonic loss and oscillatory dynamics, this regime frequently leads to faster convergence and better generalization. Existing theoretical perspectives offer complementary but incomplete insights: one identifies a self-stabilization mechanism that drives GD toward flat regions but does not yield convergence rates; another derives rigorous rates for specific losses but does not explain the underlying stabilization mechanism; and a third unifies these views for overparameterized least squares, but relies on the existence of a minimizer manifold, which is absent in classification settings. In this work, we extend this unified geometric perspective to exponential-tail losses, including logistic and cross-entropy losses in separable linear classification, where the loss infimum is not attained at any finite parameter. We identify a curvature-defined reference subspace that replaces the role of the minimizer manifold, yielding a coupled dynamical system in which orthogonal oscillations are damped by parallel progress along directions of decreasing sharpness. Under regularity conditions, which we verify for logistic and multiclass cross-entropy losses on symmetric data models, we derive explicit bounds for the iterates, not just the loss, covering both the unstable and stable regimes, and revealing a three-phase damping structure in the latter. Our analysis further uncovers a transient implicit bias induced by large step sizes: although GD converges asymptotically to the max-margin direction, large steps can leave persistent orthogonal residuals and, in certain geometries, amplify them, affecting finite-time alignment with the max-margin direction.

First seen by Oriel on .

AI summary

This work extends large-step gradient descent theory to separable classification, explaining self-stabilization, convergence rates, oscillation damping, and transient implicit bias for exponential-tail losses.

  • A curvature-defined reference subspace replaces the unavailable finite minimizer manifold in separable linear classification.
  • The resulting coupled dynamics show orthogonal oscillations being damped by progress along directions of decreasing sharpness.
  • Under verified regularity conditions for logistic and multiclass cross-entropy losses on symmetric data, the authors bound iterates in unstable and stable regimes.
  • Stable dynamics exhibit a three-phase damping structure, while large steps can leave or amplify orthogonal residuals and delay finite-time max-margin alignment.
  • Despite these transient effects, gradient descent converges asymptotically to the max-margin direction.

#gradient descent #classification #implicit bias #cross-entropy

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.17459v1https://arxiv.org/abs/2611.17459 Read discussion