Linear attention is (maybe) all you need (to understand transformer optimization)
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916158099161088 |
|---|---|
| author | Ahn, Kwangjun Cheng, Xiang Song, Minhak Yun, Chulhee Jadbabaie, Ali Sra, Suvrit |
| author_facet | Ahn, Kwangjun Cheng, Xiang Song, Minhak Yun, Chulhee Jadbabaie, Ali Sra, Suvrit |
| contents | Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_01082 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Linear attention is (maybe) all you need (to understand transformer optimization) Ahn, Kwangjun Cheng, Xiang Song, Minhak Yun, Chulhee Jadbabaie, Ali Sra, Suvrit Machine Learning Artificial Intelligence Optimization and Control Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization. |
| title | Linear attention is (maybe) all you need (to understand transformer optimization) |
| topic | Machine Learning Artificial Intelligence Optimization and Control |
| url | https://arxiv.org/abs/2310.01082 |