Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.06379 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929515405508608 |
|---|---|
| author | Koshizuka, Takeshi Fujisawa, Masahiro Tanaka, Yusuke Sato, Issei |
| author_facet | Koshizuka, Takeshi Fujisawa, Masahiro Tanaka, Yusuke Sato, Issei |
| contents | In this paper, we explores the expressivity and trainability of the Fourier Neural Operator (FNO). We establish a mean-field theory for the FNO, analyzing the behavior of the random FNO from an edge of chaos perspective. Our investigation into the expressivity of a random FNO involves examining the ordered-chaos phase transition of the network based on the weight distribution. This phase transition demonstrates characteristics unique to the FNO, induced by mode truncation, while also showcasing similarities to those of densely connected networks. Furthermore, we identify a connection between expressivity and trainability: the ordered and chaotic phases correspond to regions of vanishing and exploding gradients, respectively. This finding provides a practical prerequisite for the stable training of the FNO. Our experimental results corroborate our theoretical findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_06379 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective Koshizuka, Takeshi Fujisawa, Masahiro Tanaka, Yusuke Sato, Issei Machine Learning In this paper, we explores the expressivity and trainability of the Fourier Neural Operator (FNO). We establish a mean-field theory for the FNO, analyzing the behavior of the random FNO from an edge of chaos perspective. Our investigation into the expressivity of a random FNO involves examining the ordered-chaos phase transition of the network based on the weight distribution. This phase transition demonstrates characteristics unique to the FNO, induced by mode truncation, while also showcasing similarities to those of densely connected networks. Furthermore, we identify a connection between expressivity and trainability: the ordered and chaotic phases correspond to regions of vanishing and exploding gradients, respectively. This finding provides a practical prerequisite for the stable training of the FNO. Our experimental results corroborate our theoretical findings. |
| title | Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2310.06379 |