Fokker-Planck Analysis and Invariant Laws for a Continuous-Time Stochastic Model of Adam-Type Dynamics
Fuente:
arXiv
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910094149550080 |
|---|---|
| author | Nyström, Kaj |
| author_facet | Nyström, Kaj |
| contents | We develop a continuous-time model for the long-term dynamics of adaptive stochastic optimization, focusing on bias-corrected Adam-type methods. Starting from a finite-sum setting, we identify a canonical scaling of learning rates, decay parameters, and gradient noise that yields a coupled, time-inhomogeneous stochastic differential equation for the parameters $x_t$, first-moment tracker $z_t$, and second-moment tracker $y_t$. Bias correction persists via explicit time-dependent coefficients, and the dynamics becomes asymptotically time-homogeneous. We analyze the associated Fokker-Planck equation and, under mild regularity and dissipativity assumptions on $f$, prove existence and uniqueness of invariant measures. Noise propagation is governed by $A(x)=\mathrm{Diag}(\nabla f(x))H_f(x)$. Hypoellipticity may fail on $\mathcal D_A\times\mathbb R^m\times(\mathbb R_+)^m$, where \[ \mathcal D_A=\{x\in\mathbb R^m:\exists j,\ e_j^\top A(x)=0\}\subset\{x:\det A(x)=0\}=\mathcal D_A^\dagger, \] and critical points of $f$ lie in $\mathcal D_A$. We show $\mathcal D_A^\dagger\neq\mathbb R^m$ and use this to prove exponential convergence of the Markov semigroup $μ_0P_t$ to a unique invariant measure, uniformly in $μ_0$. The proof uses a Harris-type argument, minorization on Lyapunov sublevel sets, control constructions, and hypoellipticity on $(\mathbb R^m\setminus\mathcal D_A)\times\mathbb R^m\times(\mathbb R_+)^m$. This provides a transparent continuous-time view of Adam-type dynamics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_00840 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Fokker-Planck Analysis and Invariant Laws for a Continuous-Time Stochastic Model of Adam-Type Dynamics Nyström, Kaj Analysis of PDEs We develop a continuous-time model for the long-term dynamics of adaptive stochastic optimization, focusing on bias-corrected Adam-type methods. Starting from a finite-sum setting, we identify a canonical scaling of learning rates, decay parameters, and gradient noise that yields a coupled, time-inhomogeneous stochastic differential equation for the parameters $x_t$, first-moment tracker $z_t$, and second-moment tracker $y_t$. Bias correction persists via explicit time-dependent coefficients, and the dynamics becomes asymptotically time-homogeneous. We analyze the associated Fokker-Planck equation and, under mild regularity and dissipativity assumptions on $f$, prove existence and uniqueness of invariant measures. Noise propagation is governed by $A(x)=\mathrm{Diag}(\nabla f(x))H_f(x)$. Hypoellipticity may fail on $\mathcal D_A\times\mathbb R^m\times(\mathbb R_+)^m$, where \[ \mathcal D_A=\{x\in\mathbb R^m:\exists j,\ e_j^\top A(x)=0\}\subset\{x:\det A(x)=0\}=\mathcal D_A^\dagger, \] and critical points of $f$ lie in $\mathcal D_A$. We show $\mathcal D_A^\dagger\neq\mathbb R^m$ and use this to prove exponential convergence of the Markov semigroup $μ_0P_t$ to a unique invariant measure, uniformly in $μ_0$. The proof uses a Harris-type argument, minorization on Lyapunov sublevel sets, control constructions, and hypoellipticity on $(\mathbb R^m\setminus\mathcal D_A)\times\mathbb R^m\times(\mathbb R_+)^m$. This provides a transparent continuous-time view of Adam-type dynamics. |
| title | Fokker-Planck Analysis and Invariant Laws for a Continuous-Time Stochastic Model of Adam-Type Dynamics |
| topic | Analysis of PDEs |
| url | https://arxiv.org/abs/2604.00840 |