FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911135290097664 |
|---|---|
| author | Wang, Hui Liu, Shujie Meng, Lingwei Li, Jinyu Yang, Yifan Zhao, Shiwan Sun, Haiyang Liu, Yanqing Sun, Haoqin Zhou, Jiaming Lu, Yan Qin, Yong |
| author_facet | Wang, Hui Liu, Shujie Meng, Lingwei Li, Jinyu Yang, Yifan Zhao, Shiwan Sun, Haiyang Liu, Yanqing Sun, Haoqin Zhou, Jiaming Lu, Yan Qin, Yong |
| contents | To advance continuous-valued token modeling and temporal-coherence enforcement, we propose FELLE, an autoregressive model that integrates language modeling with token-wise flow matching. By leveraging the autoregressive nature of language models and the generative efficacy of flow matching, FELLE effectively predicts continuous-valued tokens (mel-spectrograms). For each continuous-valued token, FELLE modifies the general prior distribution in flow matching by incorporating information from the previous step, improving coherence and stability. Furthermore, to enhance synthesis quality, FELLE introduces a coarse-to-fine flow-matching mechanism, generating continuous-valued tokens hierarchically, conditioned on the language model's output. Experimental results demonstrate the potential of incorporating flow-matching techniques in autoregressive mel-spectrogram modeling, leading to significant improvements in TTS generation quality, as shown in https://aka.ms/felle. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11128 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching Wang, Hui Liu, Shujie Meng, Lingwei Li, Jinyu Yang, Yifan Zhao, Shiwan Sun, Haiyang Liu, Yanqing Sun, Haoqin Zhou, Jiaming Lu, Yan Qin, Yong Computation and Language Sound Audio and Speech Processing To advance continuous-valued token modeling and temporal-coherence enforcement, we propose FELLE, an autoregressive model that integrates language modeling with token-wise flow matching. By leveraging the autoregressive nature of language models and the generative efficacy of flow matching, FELLE effectively predicts continuous-valued tokens (mel-spectrograms). For each continuous-valued token, FELLE modifies the general prior distribution in flow matching by incorporating information from the previous step, improving coherence and stability. Furthermore, to enhance synthesis quality, FELLE introduces a coarse-to-fine flow-matching mechanism, generating continuous-valued tokens hierarchically, conditioned on the language model's output. Experimental results demonstrate the potential of incorporating flow-matching techniques in autoregressive mel-spectrogram modeling, leading to significant improvements in TTS generation quality, as shown in https://aka.ms/felle. |
| title | FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.11128 |