SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909575751401472 |
|---|---|
| author | Puvvada, Krishna C. Ladhak, Faisal Serrano, Santiago Akle Hsieh, Cheng-Ping Acharya, Shantanu Majumdar, Somshubra Jia, Fei Kriman, Samuel Sun, Simeng Rekesh, Dima Ginsburg, Boris |
| author_facet | Puvvada, Krishna C. Ladhak, Faisal Serrano, Santiago Akle Hsieh, Cheng-Ping Acharya, Shantanu Majumdar, Somshubra Jia, Fei Kriman, Samuel Sun, Simeng Rekesh, Dima Ginsburg, Boris |
| contents | We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE). Experiments demonstrate strong performance on sequence lengths significantly longer than the training length without the need for additional long-context training. This robust length extrapolation is achieved through our novel architecture, enhanced by a straightforward dynamic scaling of attention scores during inference. In addition, SWAN-GPT is more computationally efficient than standard GPT architectures, resulting in cheaper training and higher throughput. Further, we demonstrate that existing pre-trained decoder-only models can be efficiently converted to the SWAN architecture with minimal continued training, enabling longer contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08719 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling Puvvada, Krishna C. Ladhak, Faisal Serrano, Santiago Akle Hsieh, Cheng-Ping Acharya, Shantanu Majumdar, Somshubra Jia, Fei Kriman, Samuel Sun, Simeng Rekesh, Dima Ginsburg, Boris Computation and Language We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE). Experiments demonstrate strong performance on sequence lengths significantly longer than the training length without the need for additional long-context training. This robust length extrapolation is achieved through our novel architecture, enhanced by a straightforward dynamic scaling of attention scores during inference. In addition, SWAN-GPT is more computationally efficient than standard GPT architectures, resulting in cheaper training and higher throughput. Further, we demonstrate that existing pre-trained decoder-only models can be efficiently converted to the SWAN architecture with minimal continued training, enabling longer contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner. |
| title | SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.08719 |