Sharpness-Aware Teleportation on Riemannian Manifolds
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908397921632256 |
|---|---|
| author | Truong, Tuan Nguyen, Hoang-Phi Luo, Haocheng Pham, Tung Harandi, Mehrtash Phung, Dinh Le, Trung |
| author_facet | Truong, Tuan Nguyen, Hoang-Phi Luo, Haocheng Pham, Tung Harandi, Mehrtash Phung, Dinh Le, Trung |
| contents | Recent studies highlight the effectiveness of flat minima in enhancing generalization, with sharpness-aware minimization (SAM) achieving state-of-the-art performance. Additionally, insights into the intrinsic geometry of the loss landscape have shown promise for improving model generalization. Building on these advancements, we introduce a novel sharpness-aware, geometry-aware teleportation mechanism to further enhance robustness and generalization. The core innovation of our approach is to decompose each iteration into a teleportation step within a local orbit and a sharpness-aware step that transitions between different orbits, leveraging the Riemannian quotient manifold. Our approach is grounded in a theoretical framework that analyzes the generalization gap between population loss and worst-case empirical loss within the context of Riemannian manifolds. To demonstrate the effectiveness of our method, we evaluate and compare our algorithm on diverse vision benchmarks with various datasets and Riemannian manifolds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_17215 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Sharpness-Aware Teleportation on Riemannian Manifolds Truong, Tuan Nguyen, Hoang-Phi Luo, Haocheng Pham, Tung Harandi, Mehrtash Phung, Dinh Le, Trung Machine Learning Artificial Intelligence Recent studies highlight the effectiveness of flat minima in enhancing generalization, with sharpness-aware minimization (SAM) achieving state-of-the-art performance. Additionally, insights into the intrinsic geometry of the loss landscape have shown promise for improving model generalization. Building on these advancements, we introduce a novel sharpness-aware, geometry-aware teleportation mechanism to further enhance robustness and generalization. The core innovation of our approach is to decompose each iteration into a teleportation step within a local orbit and a sharpness-aware step that transitions between different orbits, leveraging the Riemannian quotient manifold. Our approach is grounded in a theoretical framework that analyzes the generalization gap between population loss and worst-case empirical loss within the context of Riemannian manifolds. To demonstrate the effectiveness of our method, we evaluate and compare our algorithm on diverse vision benchmarks with various datasets and Riemannian manifolds. |
| title | Sharpness-Aware Teleportation on Riemannian Manifolds |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2309.17215 |