Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2408.11804 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916364308971520 |
|---|---|
| author | Yunis, David Patel, Kumar Kshitij Wheeler, Samuel Savarese, Pedro Vardi, Gal Livescu, Karen Maire, Michael Walter, Matthew R. |
| author_facet | Yunis, David Patel, Kumar Kshitij Wheeler, Samuel Savarese, Pedro Vardi, Gal Livescu, Karen Maire, Michael Walter, Matthew R. |
| contents | We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale ``grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers. We also demonstrate that weight decay enhances this bias beyond its role as a norm regularizer, even in practical systems. Moreover, we show that these spectral dynamics distinguish memorizing networks from generalizing ones, offering a novel perspective on this longstanding conundrum. Additionally, we leverage spectral dynamics to explore the emergence of well-performing sparse subnetworks (lottery tickets) and the structure of the loss surface through linear mode connectivity. Our findings suggest that spectral dynamics provide a coherent framework to better understand the behavior of neural networks across diverse settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11804 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Approaching Deep Learning through the Spectral Dynamics of Weights Yunis, David Patel, Kumar Kshitij Wheeler, Samuel Savarese, Pedro Vardi, Gal Livescu, Karen Maire, Michael Walter, Matthew R. Machine Learning Artificial Intelligence We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale ``grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers. We also demonstrate that weight decay enhances this bias beyond its role as a norm regularizer, even in practical systems. Moreover, we show that these spectral dynamics distinguish memorizing networks from generalizing ones, offering a novel perspective on this longstanding conundrum. Additionally, we leverage spectral dynamics to explore the emergence of well-performing sparse subnetworks (lottery tickets) and the structure of the loss surface through linear mode connectivity. Our findings suggest that spectral dynamics provide a coherent framework to better understand the behavior of neural networks across diverse settings. |
| title | Approaching Deep Learning through the Spectral Dynamics of Weights |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2408.11804 |