Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915166322425856 |
|---|---|
| author | Bick, Aviv Katsch, Tobias Sohoni, Nimit Desai, Arjun Gu, Albert |
| author_facet | Bick, Aviv Katsch, Tobias Sohoni, Nimit Desai, Arjun Gu, Albert |
| contents | We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Furthermore, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK (Bick et al., 2024), achieving these results with less than 0.1% of the training data typically used for models of similar size. To take full advantage of their efficiency, we provide an optimized implementation of Llamba for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers. Overall, Llamba improves the tradeoff between speed, memory efficiency, and performance, making high-quality language models more accessible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14458 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing Bick, Aviv Katsch, Tobias Sohoni, Nimit Desai, Arjun Gu, Albert Machine Learning Artificial Intelligence We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Furthermore, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK (Bick et al., 2024), achieving these results with less than 0.1% of the training data typically used for models of similar size. To take full advantage of their efficiency, we provide an optimized implementation of Llamba for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers. Overall, Llamba improves the tradeoff between speed, memory efficiency, and performance, making high-quality language models more accessible. |
| title | Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.14458 |