Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.06463 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912814481801216 |
|---|---|
| author | Ma, Xuezhe Wen, Shicheng Jin, Linghao Acun, Bilge Lai, Ruihang Hou, Bohan Lin, Will Zhang, Hao Yang, Songlin Lee, Ryan Wu, Mengxi May, Jonathan Zettlemoyer, Luke Wu, Carole-Jean |
| author_facet | Ma, Xuezhe Wen, Shicheng Jin, Linghao Acun, Bilge Lai, Ruihang Hou, Bohan Lin, Will Zhang, Hao Yang, Songlin Lee, Ryan Wu, Mengxi May, Jonathan Zettlemoyer, Luke Wu, Carole-Jean |
| contents | Designing a unified neural network to efficiently and inherently process sequential data with arbitrary lengths is a central and challenging problem in sequence modeling. The design choices in Transformer, including quadratic complexity and weak length extrapolation, have limited their ability to scale to long sequences. In this work, we propose Gecko, a neural architecture that inherits the design of Mega and Megalodon (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability to capture long range dependencies, including timestep decay normalization, sliding chunk attention mechanism, and adaptive working memory. In a controlled pretraining comparison with Llama2 and Megalodon in the scale of 7 billion parameters and 2 trillion training tokens, Gecko achieves better efficiency and long-context scalability. Gecko reaches a training loss of 1.68, significantly outperforming Llama2-7B (1.75) and Megalodon-7B (1.70), and landing close to Llama2-13B (1.67). Notably, without relying on any context-extension techniques, Gecko exhibits inherent long-context processing and retrieval capabilities, stably handling sequences of up to 4 million tokens and retrieving information from contexts up to $4\times$ longer than its attention window. Code: https://github.com/XuezheMax/gecko-llm |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_06463 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths Ma, Xuezhe Wen, Shicheng Jin, Linghao Acun, Bilge Lai, Ruihang Hou, Bohan Lin, Will Zhang, Hao Yang, Songlin Lee, Ryan Wu, Mengxi May, Jonathan Zettlemoyer, Luke Wu, Carole-Jean Machine Learning Computation and Language Designing a unified neural network to efficiently and inherently process sequential data with arbitrary lengths is a central and challenging problem in sequence modeling. The design choices in Transformer, including quadratic complexity and weak length extrapolation, have limited their ability to scale to long sequences. In this work, we propose Gecko, a neural architecture that inherits the design of Mega and Megalodon (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability to capture long range dependencies, including timestep decay normalization, sliding chunk attention mechanism, and adaptive working memory. In a controlled pretraining comparison with Llama2 and Megalodon in the scale of 7 billion parameters and 2 trillion training tokens, Gecko achieves better efficiency and long-context scalability. Gecko reaches a training loss of 1.68, significantly outperforming Llama2-7B (1.75) and Megalodon-7B (1.70), and landing close to Llama2-13B (1.67). Notably, without relying on any context-extension techniques, Gecko exhibits inherent long-context processing and retrieval capabilities, stably handling sequences of up to 4 million tokens and retrieving information from contexts up to $4\times$ longer than its attention window. Code: https://github.com/XuezheMax/gecko-llm |
| title | Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.06463 |