Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909642074882048 |
|---|---|
| author | Park, Jaesung R. Suh, Jaewook J. Hong, Youngjoon Ryu, Ernest K. |
| author_facet | Park, Jaesung R. Suh, Jaewook J. Hong, Youngjoon Ryu, Ernest K. |
| contents | In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08595 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models Park, Jaesung R. Suh, Jaewook J. Hong, Youngjoon Ryu, Ernest K. Numerical Analysis Machine Learning In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions. |
| title | Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models |
| topic | Numerical Analysis Machine Learning |
| url | https://arxiv.org/abs/2412.08595 |