Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.03717 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916383641567232 |
|---|---|
| author | Lovelace, Justin Ray, Soham Kim, Kwangyoun Weinberger, Kilian Q. Wu, Felix |
| author_facet | Lovelace, Justin Ray, Soham Kim, Kwangyoun Weinberger, Kilian Q. Wu, Felix |
| contents | This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03717 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sample-Efficient Diffusion for Text-To-Speech Synthesis Lovelace, Justin Ray, Soham Kim, Kwangyoun Weinberger, Kilian Q. Wu, Felix Sound Artificial Intelligence Machine Learning This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data. |
| title | Sample-Efficient Diffusion for Text-To-Speech Synthesis |
| topic | Sound Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.03717 |