Researchers reveal how stochastic noise affects training dynamics in deep linear networks during saddle-point transitions

Study analyzes stochastic gradient descent (SGD) behavior in deep linear networks (DLNs), a simplified model for understanding real neural network training

Researchers modeled SGD dynamics as Langevin equations with anisotropic, state-dependent noise to explain the saddle-to-saddle training regime

Key finding: maximum diffusion along each mode occurs before the corresponding feature is fully learned, providing new insight into feature learning order

Under aligned and balanced weight assumptions, the team derived exact decompositions into one-dimensional stochastic differential equations for each mode

Analysis of stationary distributions shows SGD behavior varies based on label noise presence, advancing theoretical understanding of SGD's implicit regularization