Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.20094 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915758660911104 |
|---|---|
| author | Wu, Haibin Do, Bach Viet Suda, Naveen Chan, Julian R, Madhavan C Yang, Gene-Ping Wu, Yi-Chiao Kanda, Naoyuki Adi, Yossef Lei, Xin Liu, Yue Metze, Florian Liu, Yuzong |
| author_facet | Wu, Haibin Do, Bach Viet Suda, Naveen Chan, Julian R, Madhavan C Yang, Gene-Ping Wu, Yi-Chiao Kanda, Naoyuki Adi, Yossef Lei, Xin Liu, Yue Metze, Florian Liu, Yuzong |
| contents | Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20094 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS Wu, Haibin Do, Bach Viet Suda, Naveen Chan, Julian R, Madhavan C Yang, Gene-Ping Wu, Yi-Chiao Kanda, Naoyuki Adi, Yossef Lei, Xin Liu, Yue Metze, Florian Liu, Yuzong Audio and Speech Processing Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality. |
| title | T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2601.20094 |