Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910604120293376 |
|---|---|
| author | Ren, Yi Sutherland, Danica J. |
| author_facet | Ren, Yi Sutherland, Danica J. |
| contents | Obtaining compositional mappings is important for the model to generalize well compositionally. To better understand when and how to encourage the model to learn such mappings, we study their uniqueness through different perspectives. Specifically, we first show that the compositional mappings are the simplest bijections through the lens of coding length (i.e., an upper bound of their Kolmogorov complexity). This property explains why models having such mappings can generalize well. We further show that the simplicity bias is usually an intrinsic property of neural network training via gradient descent. That partially explains why some models spontaneously generalize well when they are trained appropriately. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09626 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics Ren, Yi Sutherland, Danica J. Machine Learning Artificial Intelligence Obtaining compositional mappings is important for the model to generalize well compositionally. To better understand when and how to encourage the model to learn such mappings, we study their uniqueness through different perspectives. Specifically, we first show that the compositional mappings are the simplest bijections through the lens of coding length (i.e., an upper bound of their Kolmogorov complexity). This property explains why models having such mappings can generalize well. We further show that the simplicity bias is usually an intrinsic property of neural network training via gradient descent. That partially explains why some models spontaneously generalize well when they are trained appropriately. |
| title | Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2409.09626 |