Intra-Layer Recurrence in Transformers for Language Modeling
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915303065124864 |
|---|---|
| author | Nguyen, Anthony Lin, Wenjun |
| author_facet | Nguyen, Anthony Lin, Wenjun |
| contents | Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01855 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Intra-Layer Recurrence in Transformers for Language Modeling Nguyen, Anthony Lin, Wenjun Computation and Language Artificial Intelligence Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures. |
| title | Intra-Layer Recurrence in Transformers for Language Modeling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.01855 |