DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929336728158208 |
|---|---|
| author | Hao, Zhongkai Su, Chang Liu, Songming Berner, Julius Ying, Chengyang Su, Hang Anandkumar, Anima Song, Jian Zhu, Jun |
| author_facet | Hao, Zhongkai Su, Chang Liu, Songming Berner, Julius Ying, Chengyang Su, Hang Anandkumar, Anima Song, Jian Zhu, Jun |
| contents | Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_03542 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training Hao, Zhongkai Su, Chang Liu, Songming Berner, Julius Ying, Chengyang Su, Hang Anandkumar, Anima Song, Jian Zhu, Jun Machine Learning Numerical Analysis Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}. |
| title | DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2403.03542 |