Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.03187 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913223235600384 |
|---|---|
| author | Lion, Kai Noci, Lorenzo Hofmann, Thomas Bachmann, Gregor |
| author_facet | Lion, Kai Noci, Lorenzo Hofmann, Thomas Bachmann, Gregor |
| contents | The multi-modal nature of neural loss landscapes is often considered to be the main driver behind the empirical success of deep ensembles. In this work, we probe this belief by constructing various "connected" ensembles which are restricted to lie in the same basin. Through our experiments, we demonstrate that increased connectivity indeed negatively impacts performance. However, when incorporating the knowledge from other basins implicitly through distillation, we show that the gap in performance can be mitigated by re-discovering (multi-basin) deep ensembles within a single basin. Thus, we conjecture that while the extra-basin knowledge is at least partially present in any given basin, it cannot be easily harnessed without learning it from other basins. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03187 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How Good is a Single Basin? Lion, Kai Noci, Lorenzo Hofmann, Thomas Bachmann, Gregor Machine Learning The multi-modal nature of neural loss landscapes is often considered to be the main driver behind the empirical success of deep ensembles. In this work, we probe this belief by constructing various "connected" ensembles which are restricted to lie in the same basin. Through our experiments, we demonstrate that increased connectivity indeed negatively impacts performance. However, when incorporating the knowledge from other basins implicitly through distillation, we show that the gap in performance can be mitigated by re-discovering (multi-basin) deep ensembles within a single basin. Thus, we conjecture that while the extra-basin knowledge is at least partially present in any given basin, it cannot be easily harnessed without learning it from other basins. |
| title | How Good is a Single Basin? |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.03187 |