Variational Learning is Effective for Large Deep Networks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913379692576768 |
|---|---|
| author | Shen, Yuesong Daheim, Nico Cong, Bai Nickl, Peter Marconi, Gian Maria Bazan, Clement Yokota, Rio Gurevych, Iryna Cremers, Daniel Khan, Mohammad Emtiyaz Möllenhoff, Thomas |
| author_facet | Shen, Yuesong Daheim, Nico Cong, Bai Nickl, Peter Marconi, Gian Maria Bazan, Clement Yokota, Rio Gurevych, Iryna Cremers, Daniel Khan, Mohammad Emtiyaz Möllenhoff, Thomas |
| contents | We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve finetuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data. We find overwhelming evidence that variational learning is effective. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17641 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Variational Learning is Effective for Large Deep Networks Shen, Yuesong Daheim, Nico Cong, Bai Nickl, Peter Marconi, Gian Maria Bazan, Clement Yokota, Rio Gurevych, Iryna Cremers, Daniel Khan, Mohammad Emtiyaz Möllenhoff, Thomas Machine Learning Artificial Intelligence Computation and Language Optimization and Control We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve finetuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data. We find overwhelming evidence that variational learning is effective. |
| title | Variational Learning is Effective for Large Deep Networks |
| topic | Machine Learning Artificial Intelligence Computation and Language Optimization and Control |
| url | https://arxiv.org/abs/2402.17641 |