Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.00563 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910036548124672 |
|---|---|
| author | Zhang, Sen Wei, Jianguo Lu, Wenhuan Yue, Xianghu Li, Wei Li, Qiang Zhao, Pengcheng Cai, Ming Si, Luo |
| author_facet | Zhang, Sen Wei, Jianguo Lu, Wenhuan Yue, Xianghu Li, Wei Li, Qiang Zhao, Pengcheng Cai, Ming Si, Luo |
| contents | The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00563 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion Zhang, Sen Wei, Jianguo Lu, Wenhuan Yue, Xianghu Li, Wei Li, Qiang Zhao, Pengcheng Cai, Ming Si, Luo Sound Artificial Intelligence The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy. |
| title | Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2603.00563 |