Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908280920473600 |
|---|---|
| author | Thiombiano, Abdoul Majid O. Hnich, Brahim Mrad, Ali Ben Mkaouer, Mohamed Wiem |
| author_facet | Thiombiano, Abdoul Majid O. Hnich, Brahim Mrad, Ali Ben Mkaouer, Mohamed Wiem |
| contents | The current era of Natural Language Processing (NLP) is dominated by Transformer models. However, novel architectures relying on recurrent mechanisms, such as xLSTM and Mamba, have been proposed as alternatives to attention-based models. Although computation is done differently than with the attention mechanism mechanism, these recurrent models yield good results and sometimes even outperform state-of-the-art attention-based models. In this work, we propose Distil-xLSTM, an xLSTM-based Small Language Model (SLM) trained by distilling knowledge from a Large Language Model (LLM) that shows promising results while being compute and scale efficient. Our Distil-xLSTM focuses on approximating a transformer-based model attention parametrization using its recurrent sequence mixing components and shows good results with minimal training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_18565 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures Thiombiano, Abdoul Majid O. Hnich, Brahim Mrad, Ali Ben Mkaouer, Mohamed Wiem Machine Learning Artificial Intelligence Computation and Language The current era of Natural Language Processing (NLP) is dominated by Transformer models. However, novel architectures relying on recurrent mechanisms, such as xLSTM and Mamba, have been proposed as alternatives to attention-based models. Although computation is done differently than with the attention mechanism mechanism, these recurrent models yield good results and sometimes even outperform state-of-the-art attention-based models. In this work, we propose Distil-xLSTM, an xLSTM-based Small Language Model (SLM) trained by distilling knowledge from a Large Language Model (LLM) that shows promising results while being compute and scale efficient. Our Distil-xLSTM focuses on approximating a transformer-based model attention parametrization using its recurrent sequence mixing components and shows good results with minimal training. |
| title | Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.18565 |