Wedi'i Gadw mewn:
| Prif Awduron: | , , |
|---|---|
| Fformat: | Preprint |
| Cyhoeddwyd: |
2024
|
| Pynciau: | |
| Mynediad Ar-lein: | https://arxiv.org/abs/2402.03021 |
| Tagiau: |
Ychwanegu Tag
Dim Tagiau, Byddwch y cyntaf i dagio'r cofnod hwn!
|
| _version_ | 1866917583396012032 |
|---|---|
| author | He, Juncai Liu, Liangchen Tsai, Yen-Hsi Richard |
| author_facet | He, Juncai Liu, Liangchen Tsai, Yen-Hsi Richard |
| contents | This paper investigates the impact of multiscale data on machine learning algorithms, particularly in the context of deep learning. A dataset is multiscale if its distribution shows large variations in scale across different directions. This paper reveals multiscale structures in the loss landscape, including its gradients and Hessians inherited from the data. Correspondingly, it introduces a novel gradient descent approach, drawing inspiration from multiscale algorithms used in scientific computing. This approach seeks to transcend empirical learning rate selection, offering a more systematic, data-informed strategy to enhance training efficiency, especially in the later stages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03021 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Data-induced multiscale losses and efficient multirate gradient descent schemes He, Juncai Liu, Liangchen Tsai, Yen-Hsi Richard Machine Learning Numerical Analysis 65F10, 65F45, 68T07 G.1.6; I.2.6 This paper investigates the impact of multiscale data on machine learning algorithms, particularly in the context of deep learning. A dataset is multiscale if its distribution shows large variations in scale across different directions. This paper reveals multiscale structures in the loss landscape, including its gradients and Hessians inherited from the data. Correspondingly, it introduces a novel gradient descent approach, drawing inspiration from multiscale algorithms used in scientific computing. This approach seeks to transcend empirical learning rate selection, offering a more systematic, data-informed strategy to enhance training efficiency, especially in the later stages. |
| title | Data-induced multiscale losses and efficient multirate gradient descent schemes |
| topic | Machine Learning Numerical Analysis 65F10, 65F45, 68T07 G.1.6; I.2.6 |
| url | https://arxiv.org/abs/2402.03021 |