Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.15647 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918016350945280 |
|---|---|
| author | Fan, Ying Du, Yilun Ramchandran, Kannan Lee, Kangwook |
| author_facet | Fan, Ying Du, Yilun Ramchandran, Kannan Lee, Kangwook |
| contents | Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15647 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Looped Transformers for Length Generalization Fan, Ying Du, Yilun Ramchandran, Kannan Lee, Kangwook Machine Learning Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks. |
| title | Looped Transformers for Length Generalization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2409.15647 |