Effective Distillation to Hybrid xLSTM Architectures
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915867511488512 |
|---|---|
| author | Hauzenberger, Lukas Schmidinger, Niklas Schmied, Thomas Hartl, Anamaria-Roberta Stap, David Hoedt, Pieter-Jan Beck, Maximilian Böck, Sebastian Klambauer, Günter Hochreiter, Sepp |
| author_facet | Hauzenberger, Lukas Schmidinger, Niklas Schmied, Thomas Hartl, Anamaria-Roberta Stap, David Hoedt, Pieter-Jan Beck, Maximilian Böck, Sebastian Klambauer, Günter Hochreiter, Sepp |
| contents | There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15590 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Effective Distillation to Hybrid xLSTM Architectures Hauzenberger, Lukas Schmidinger, Niklas Schmied, Thomas Hartl, Anamaria-Roberta Stap, David Hoedt, Pieter-Jan Beck, Maximilian Böck, Sebastian Klambauer, Günter Hochreiter, Sepp Machine Learning There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs. |
| title | Effective Distillation to Hybrid xLSTM Architectures |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2603.15590 |