Multiple Physics Pretraining for Physical Surrogate Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909423901868032 |
|---|---|
| author | McCabe, Michael Blancard, Bruno Régaldo-Saint Parker, Liam Holden Ohana, Ruben Cranmer, Miles Bietti, Alberto Eickenberg, Michael Golkar, Siavash Krawezik, Geraud Lanusse, Francois Pettee, Mariel Tesileanu, Tiberiu Cho, Kyunghyun Ho, Shirley |
| author_facet | McCabe, Michael Blancard, Bruno Régaldo-Saint Parker, Liam Holden Ohana, Ruben Cranmer, Miles Bietti, Alberto Eickenberg, Michael Golkar, Siavash Krawezik, Geraud Lanusse, Francois Pettee, Mariel Tesileanu, Tiberiu Cho, Kyunghyun Ho, Shirley |
| contents | We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02994 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Multiple Physics Pretraining for Physical Surrogate Models McCabe, Michael Blancard, Bruno Régaldo-Saint Parker, Liam Holden Ohana, Ruben Cranmer, Miles Bietti, Alberto Eickenberg, Michael Golkar, Siavash Krawezik, Geraud Lanusse, Francois Pettee, Mariel Tesileanu, Tiberiu Cho, Kyunghyun Ho, Shirley Machine Learning Artificial Intelligence We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility. |
| title | Multiple Physics Pretraining for Physical Surrogate Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.02994 |