MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918304277331968 |
|---|---|
| author | An, Keyu Zhang, Zhiyu Gao, Changfeng Li, Yabin Peng, Zhendong Wang, Haoxu Du, Zhihao Zhao, Han Gao, Zhifu Li, Xiangang |
| author_facet | An, Keyu Zhang, Zhiyu Gao, Changfeng Li, Yabin Peng, Zhendong Wang, Haoxu Du, Zhihao Zhao, Han Gao, Zhifu Li, Xiangang |
| contents | This work introduces MELA-TTS, a novel joint transformer-diffusion framework for end-to-end text-to-speech synthesis. By autoregressively generating continuous mel-spectrogram frames from linguistic and speaker conditions, our architecture eliminates the need for speech tokenization and multi-stage processing pipelines. To address the inherent difficulties of modeling continuous features, we propose a representation alignment module that aligns output representations of the transformer decoder with semantic embeddings from a pretrained ASR encoder during training. This mechanism not only speeds up training convergence, but also enhances cross-modal coherence between the textual and acoustic domains. Comprehensive experiments demonstrate that MELA-TTS achieves state-of-the-art performance across multiple evaluation metrics while maintaining robust zero-shot voice cloning capabilities, in both offline and streaming synthesis modes. Our results establish a new benchmark for continuous feature generation approaches in TTS, offering a compelling alternative to discrete-token-based paradigms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_14784 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis An, Keyu Zhang, Zhiyu Gao, Changfeng Li, Yabin Peng, Zhendong Wang, Haoxu Du, Zhihao Zhao, Han Gao, Zhifu Li, Xiangang Audio and Speech Processing This work introduces MELA-TTS, a novel joint transformer-diffusion framework for end-to-end text-to-speech synthesis. By autoregressively generating continuous mel-spectrogram frames from linguistic and speaker conditions, our architecture eliminates the need for speech tokenization and multi-stage processing pipelines. To address the inherent difficulties of modeling continuous features, we propose a representation alignment module that aligns output representations of the transformer decoder with semantic embeddings from a pretrained ASR encoder during training. This mechanism not only speeds up training convergence, but also enhances cross-modal coherence between the textual and acoustic domains. Comprehensive experiments demonstrate that MELA-TTS achieves state-of-the-art performance across multiple evaluation metrics while maintaining robust zero-shot voice cloning capabilities, in both offline and streaming synthesis modes. Our results establish a new benchmark for continuous feature generation approaches in TTS, offering a compelling alternative to discrete-token-based paradigms. |
| title | MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.14784 |