Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.20763 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915176551284736 |
|---|---|
| author | Tan, Zhiquan Huang, Weiran |
| author_facet | Tan, Zhiquan Huang, Weiran |
| contents | The interplay of optimizers and architectures in neural networks is complicated and hard to understand why some optimizers work better on some specific architectures. In this paper, we find that the traditionally used sharpness metric does not fully explain the intricate interplay and introduces information-theoretic metrics called entropy gap to better help analyze. It is found that both sharpness and entropy gap affect the performance, including the optimization dynamic and generalization. We further use information-theoretic tools to understand a recently proposed optimizer called Lion and find ways to improve it. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20763 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Information-Theoretic Perspectives on Optimizers Tan, Zhiquan Huang, Weiran Machine Learning The interplay of optimizers and architectures in neural networks is complicated and hard to understand why some optimizers work better on some specific architectures. In this paper, we find that the traditionally used sharpness metric does not fully explain the intricate interplay and introduces information-theoretic metrics called entropy gap to better help analyze. It is found that both sharpness and entropy gap affect the performance, including the optimization dynamic and generalization. We further use information-theoretic tools to understand a recently proposed optimizer called Lion and find ways to improve it. |
| title | Information-Theoretic Perspectives on Optimizers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.20763 |