Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.21101 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910174036361216 |
|---|---|
| author | Kinch, Brooks Hu, Xiaozhe Huang, Yilong Hansen, Martine Dyring Meltzer, Sunniva Hamlin, Nathaniel Donald Sirajuddin, David Cyr, Eric C. Trask, Nathaniel |
| author_facet | Kinch, Brooks Hu, Xiaozhe Huang, Yilong Hansen, Martine Dyring Meltzer, Sunniva Hamlin, Nathaniel Donald Sirajuddin, David Cyr, Eric C. Trask, Nathaniel |
| contents | For autoregressive modeling of chaotic dynamical systems over long time horizons, the stability of both training and inference is a major challenge in building scientific foundation models. We present a hybrid technique in which an autoregressive transformer is embedded within a novel shooting-based mixed finite element scheme, exposing topological structure that enables provable stability. For forward problems, we prove preservation of discrete energies, while for training we prove uniform bounds on gradients, provably avoiding the exploding gradient problem. Combined with a vision transformer, this yields latent tokens admitting structure-preserving dynamics. We outperform modern foundation models with a $65\times$ reduction in model parameters and long-horizon forecasting of chaotic systems. A "mini-foundation" model of a fusion component shows that 12 simulations suffice to train a real-time surrogate, achieving a $9{,}000\times$ speedup over particle-in-cell simulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21101 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Hybridizable Neural Time Integrator for Stable Autoregressive Forecasting Kinch, Brooks Hu, Xiaozhe Huang, Yilong Hansen, Martine Dyring Meltzer, Sunniva Hamlin, Nathaniel Donald Sirajuddin, David Cyr, Eric C. Trask, Nathaniel Machine Learning Numerical Analysis For autoregressive modeling of chaotic dynamical systems over long time horizons, the stability of both training and inference is a major challenge in building scientific foundation models. We present a hybrid technique in which an autoregressive transformer is embedded within a novel shooting-based mixed finite element scheme, exposing topological structure that enables provable stability. For forward problems, we prove preservation of discrete energies, while for training we prove uniform bounds on gradients, provably avoiding the exploding gradient problem. Combined with a vision transformer, this yields latent tokens admitting structure-preserving dynamics. We outperform modern foundation models with a $65\times$ reduction in model parameters and long-horizon forecasting of chaotic systems. A "mini-foundation" model of a fusion component shows that 12 simulations suffice to train a real-time surrogate, achieving a $9{,}000\times$ speedup over particle-in-cell simulation. |
| title | A Hybridizable Neural Time Integrator for Stable Autoregressive Forecasting |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2604.21101 |