Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.12487 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913625802801152 |
|---|---|
| author | Xiao, Zipeng Hao, Zhongkai Lin, Bokai Deng, Zhijie Su, Hang |
| author_facet | Xiao, Zipeng Hao, Zhongkai Lin, Bokai Deng, Zhijie Su, Hang |
| contents | Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_12487 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Improved Operator Learning by Orthogonal Attention Xiao, Zipeng Hao, Zhongkai Lin, Bokai Deng, Zhijie Su, Hang Machine Learning Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins. |
| title | Improved Operator Learning by Orthogonal Attention |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2310.12487 |