Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.12996 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916328792653824 |
|---|---|
| author | Lu, Haiquan Liu, Xiaotian Zhou, Yefan Li, Qunli Keutzer, Kurt Mahoney, Michael W. Yan, Yujun Yang, Huanrui Yang, Yaoqing |
| author_facet | Lu, Haiquan Liu, Xiaotian Zhou, Yefan Li, Qunli Keutzer, Kurt Mahoney, Michael W. Yan, Yujun Yang, Huanrui Yang, Yaoqing |
| contents | Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12996 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance Lu, Haiquan Liu, Xiaotian Zhou, Yefan Li, Qunli Keutzer, Kurt Mahoney, Michael W. Yan, Yujun Yang, Huanrui Yang, Yaoqing Machine Learning Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios. |
| title | Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.12996 |