Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.15399 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912340367114240 |
|---|---|
| author | Zamir, Guy Dokania, Aryan Zhao, Bo Yu, Rose |
| author_facet | Zamir, Guy Dokania, Aryan Zhao, Bo Yu, Rose |
| contents | We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance optimization efficiency. Prior work has shown that jointly learning symmetry transformations and local updates improves meta-optimizer performance. Supporting this, our theoretical analysis demonstrates that even without identifying the optimal group element, the method locally resembles Newton's method. We further provide an example where the algorithm provably learns the correct symmetry transformation during training. To empirically evaluate L2O with teleportation, we introduce a benchmark, analyze its success and failure cases, and show that enhancements like momentum further improve performance. Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15399 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Learning to Optimize Using Parameter Symmetries Zamir, Guy Dokania, Aryan Zhao, Bo Yu, Rose Machine Learning We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance optimization efficiency. Prior work has shown that jointly learning symmetry transformations and local updates improves meta-optimizer performance. Supporting this, our theoretical analysis demonstrates that even without identifying the optimal group element, the method locally resembles Newton's method. We further provide an example where the algorithm provably learns the correct symmetry transformation during training. To empirically evaluate L2O with teleportation, we introduce a benchmark, analyze its success and failure cases, and show that enhancements like momentum further improve performance. Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization. |
| title | Improving Learning to Optimize Using Parameter Symmetries |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.15399 |