Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917890412773376 |
|---|---|
| author | Zhang, Zhongwang Lin, Pengxiao Wang, Zhiwei Zhang, Yaoyu Xu, Zhi-Qin John |
| author_facet | Zhang, Zhongwang Lin, Pengxiao Wang, Zhiwei Zhang, Yaoyu Xu, Zhi-Qin John |
| contents | Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the mechanisms of how transformers behave on unseen compositional tasks. We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential (reasoning-based) solutions, which capture the underlying compositional primitives, or symmetric (memory-based) solutions, which simply memorize mappings without understanding the compositional structure. By analyzing the information flow and vector representations within the model, we reveal the distinct mechanisms underlying these solution types. We further find that inferential (reasoning-based) solutions exhibit low complexity bias, which we hypothesize is a key factor enabling them to learn individual mappings for single anchors. We validate our conclusions on various real-world datasets. Our findings provide valuable insights into the role of initialization scale in tuning the reasoning and memorizing ability and we propose the initialization rate $γ$ to be a convenient tunable hyper-parameter in common deep learning frameworks, where $1/d_{\mathrm{in}}^γ$ is the standard deviation of parameters of the layer with $d_{\mathrm{in}}$ input neurons. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05409 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing Zhang, Zhongwang Lin, Pengxiao Wang, Zhiwei Zhang, Yaoyu Xu, Zhi-Qin John Machine Learning Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the mechanisms of how transformers behave on unseen compositional tasks. We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential (reasoning-based) solutions, which capture the underlying compositional primitives, or symmetric (memory-based) solutions, which simply memorize mappings without understanding the compositional structure. By analyzing the information flow and vector representations within the model, we reveal the distinct mechanisms underlying these solution types. We further find that inferential (reasoning-based) solutions exhibit low complexity bias, which we hypothesize is a key factor enabling them to learn individual mappings for single anchors. We validate our conclusions on various real-world datasets. Our findings provide valuable insights into the role of initialization scale in tuning the reasoning and memorizing ability and we propose the initialization rate $γ$ to be a convenient tunable hyper-parameter in common deep learning frameworks, where $1/d_{\mathrm{in}}^γ$ is the standard deviation of parameters of the layer with $d_{\mathrm{in}}$ input neurons. |
| title | Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.05409 |