Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.15664 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918060817907712 |
|---|---|
| author | Lim, Sungjun Yeom, Jeyoon Kim, Sooyon Byun, Hoyoon Kang, Jinho Jung, Yohan Jung, Jiyoung Song, Kyungwoo |
| author_facet | Lim, Sungjun Yeom, Jeyoon Kim, Sooyon Byun, Hoyoon Kang, Jinho Jung, Yohan Jung, Jiyoung Song, Kyungwoo |
| contents | Bayesian neural networks (BNNs) estimate the posterior distribution of model parameters and utilize posterior samples for Bayesian Model Averaging (BMA) in prediction. However, despite the crucial role of flatness in the loss landscape in improving the generalization of neural networks, its impact on BMA has been largely overlooked. In this work, we explore how posterior flatness influences BMA generalization and empirically demonstrate that (1) most approximate Bayesian inference methods fail to yield a flat posterior and (2) BMA predictions, without considering posterior flatness, are less effective at improving generalization. To address this, we propose Flat Posterior-aware Bayesian Model Averaging (FP-BMA), a novel training objective that explicitly encourages flat posteriors in a principled Bayesian manner. We also introduce a Flat Posterior-aware Bayesian Transfer Learning scheme that enhances generalization in downstream tasks. Empirically, we show that FP-BMA successfully captures flat posteriors, improving generalization performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15664 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Flat Posterior Does Matter For Bayesian Model Averaging Lim, Sungjun Yeom, Jeyoon Kim, Sooyon Byun, Hoyoon Kang, Jinho Jung, Yohan Jung, Jiyoung Song, Kyungwoo Machine Learning Bayesian neural networks (BNNs) estimate the posterior distribution of model parameters and utilize posterior samples for Bayesian Model Averaging (BMA) in prediction. However, despite the crucial role of flatness in the loss landscape in improving the generalization of neural networks, its impact on BMA has been largely overlooked. In this work, we explore how posterior flatness influences BMA generalization and empirically demonstrate that (1) most approximate Bayesian inference methods fail to yield a flat posterior and (2) BMA predictions, without considering posterior flatness, are less effective at improving generalization. To address this, we propose Flat Posterior-aware Bayesian Model Averaging (FP-BMA), a novel training objective that explicitly encourages flat posteriors in a principled Bayesian manner. We also introduce a Flat Posterior-aware Bayesian Transfer Learning scheme that enhances generalization in downstream tasks. Empirically, we show that FP-BMA successfully captures flat posteriors, improving generalization performance. |
| title | Flat Posterior Does Matter For Bayesian Model Averaging |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.15664 |