Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.00024 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909992875982848 |
|---|---|
| author | Amarel, James Hengartner, Nicolas Miller, Robyn Singh, Kamaljeet Mansingh, Siddharth Mohan, Arvind Migliori, Benjamin Casleton, Emily Skurikhin, Alexei Lawrence, Earl Kunde, Gerd J. |
| author_facet | Amarel, James Hengartner, Nicolas Miller, Robyn Singh, Kamaljeet Mansingh, Siddharth Mohan, Arvind Migliori, Benjamin Casleton, Emily Skurikhin, Alexei Lawrence, Earl Kunde, Gerd J. |
| contents | Foundation models trained as autoregressive PDE surrogates hold significant promise for accelerating scientific discovery through their capacity to both extrapolate beyond training regimes and efficiently adapt to downstream tasks despite a paucity of examples for fine-tuning. However, reliably achieving genuine generalization - a necessary capability for producing novel scientific insights and robustly performing during deployment - remains a critical challenge. Establishing whether or not these requirements are met demands evaluation metrics capable of clearly distinguishing genuine model generalization from mere memorization.
We apply the influence function formalism to systematically characterize how autoregressive PDE surrogates assimilate and propagate information derived from diverse physical scenarios, revealing fundamental limitations of standard models and training routines in addition to providing actionable insights regarding the design of improved surrogates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00024 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherence Amarel, James Hengartner, Nicolas Miller, Robyn Singh, Kamaljeet Mansingh, Siddharth Mohan, Arvind Migliori, Benjamin Casleton, Emily Skurikhin, Alexei Lawrence, Earl Kunde, Gerd J. Computational Physics Machine Learning Foundation models trained as autoregressive PDE surrogates hold significant promise for accelerating scientific discovery through their capacity to both extrapolate beyond training regimes and efficiently adapt to downstream tasks despite a paucity of examples for fine-tuning. However, reliably achieving genuine generalization - a necessary capability for producing novel scientific insights and robustly performing during deployment - remains a critical challenge. Establishing whether or not these requirements are met demands evaluation metrics capable of clearly distinguishing genuine model generalization from mere memorization. We apply the influence function formalism to systematically characterize how autoregressive PDE surrogates assimilate and propagate information derived from diverse physical scenarios, revealing fundamental limitations of standard models and training routines in addition to providing actionable insights regarding the design of improved surrogates. |
| title | Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherence |
| topic | Computational Physics Machine Learning |
| url | https://arxiv.org/abs/2509.00024 |