TransMLA: Multi-Head Latent Attention Is All You Need
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913889857306624 |
|---|---|
| author | Meng, Fanxu Tang, Pingzhi Tang, Xiaojuan Yao, Zengwei Sun, Xing Zhang, Muhan |
| author_facet | Meng, Fanxu Tang, Pingzhi Tang, Xiaojuan Yao, Zengwei Sun, Xing Zhang, Muhan |
| contents | In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07864 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TransMLA: Multi-Head Latent Attention Is All You Need Meng, Fanxu Tang, Pingzhi Tang, Xiaojuan Yao, Zengwei Sun, Xing Zhang, Muhan Machine Learning Artificial Intelligence In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized. |
| title | TransMLA: Multi-Head Latent Attention Is All You Need |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.07864 |