Linear Attention for Efficient Bidirectional Sequence Modeling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911184438951936 |
|---|---|
| author | Afzal, Arshia Rocamora, Elias Abad Candogan, Leyla Naz Puigdemont, Pol Tonin, Francesco Wu, Yongtao Shoaran, Mahsa Cevher, Volkan |
| author_facet | Afzal, Arshia Rocamora, Elias Abad Candogan, Leyla Naz Puigdemont, Pol Tonin, Francesco Wu, Yongtao Shoaran, Mahsa Cevher, Volkan |
| contents | Linear Transformers and State Space Models have emerged as efficient alternatives to softmax Transformers for causal sequence modeling, enabling parallel training via matrix multiplication and efficient RNN-style inference. However, despite their success in causal tasks, no unified framework exists for applying Linear Transformers to bidirectional sequence modeling. We introduce LION, the first framework to systematically extend Linear Transformers to the bidirectional setting. LION generalizes three core representations commonly used in the causal case - full Linear Attention , bidirectional RNN, and chunkwise parallel form - to the bidirectional setting. These forms are theoretically equivalent and enable models to exploit the strengths of each during training and inference. We prove that a broad class of Linear Transformers can be extended using LION and validate our framework via three core examples based on the choice of decay type: LION-LIT, the bidirectional extension of arXiv:2006.16236; LION-D, based on arXiv:2307.08621; and LION-S, a variant using selective decay arXiv:2103.02143, arXiv:2312.0075. Across standard bidirectional tasks, LION enables models to match or exceed the performance of softmax Transformers, while offering significantly faster training and more efficient inference than existing State Space Models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16249 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Linear Attention for Efficient Bidirectional Sequence Modeling Afzal, Arshia Rocamora, Elias Abad Candogan, Leyla Naz Puigdemont, Pol Tonin, Francesco Wu, Yongtao Shoaran, Mahsa Cevher, Volkan Machine Learning Artificial Intelligence Linear Transformers and State Space Models have emerged as efficient alternatives to softmax Transformers for causal sequence modeling, enabling parallel training via matrix multiplication and efficient RNN-style inference. However, despite their success in causal tasks, no unified framework exists for applying Linear Transformers to bidirectional sequence modeling. We introduce LION, the first framework to systematically extend Linear Transformers to the bidirectional setting. LION generalizes three core representations commonly used in the causal case - full Linear Attention , bidirectional RNN, and chunkwise parallel form - to the bidirectional setting. These forms are theoretically equivalent and enable models to exploit the strengths of each during training and inference. We prove that a broad class of Linear Transformers can be extended using LION and validate our framework via three core examples based on the choice of decay type: LION-LIT, the bidirectional extension of arXiv:2006.16236; LION-D, based on arXiv:2307.08621; and LION-S, a variant using selective decay arXiv:2103.02143, arXiv:2312.0075. Across standard bidirectional tasks, LION enables models to match or exceed the performance of softmax Transformers, while offering significantly faster training and more efficient inference than existing State Space Models. |
| title | Linear Attention for Efficient Bidirectional Sequence Modeling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.16249 |