A Non-autoregressive Model for Joint STT and TTS
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929682939641856 |
|---|---|
| author | Sunder, Vishal Kingsbury, Brian Saon, George Thomas, Samuel Shechtman, Slava Aronowitz, Hagai Fosler-Lussier, Eric Lastras, Luis |
| author_facet | Sunder, Vishal Kingsbury, Brian Saon, George Thomas, Samuel Shechtman, Slava Aronowitz, Hagai Fosler-Lussier, Eric Lastras, Luis |
| contents | In this paper, we take a step towards jointly modeling automatic speech recognition (STT) and speech synthesis (TTS) in a fully non-autoregressive way. We develop a novel multimodal framework capable of handling the speech and text modalities as input either individually or together. The proposed model can also be trained with unpaired speech or text data owing to its multimodal nature. We further propose an iterative refinement strategy to improve the STT and TTS performance of our model such that the partial hypothesis at the output can be fed back to the input of our model, thus iteratively improving both STT and TTS predictions. We show that our joint model can effectively perform both STT and TTS tasks, outperforming the STT-specific baseline in all tasks and performing competitively with the TTS-specific baseline across a wide range of evaluation metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09104 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Non-autoregressive Model for Joint STT and TTS Sunder, Vishal Kingsbury, Brian Saon, George Thomas, Samuel Shechtman, Slava Aronowitz, Hagai Fosler-Lussier, Eric Lastras, Luis Sound Artificial Intelligence Audio and Speech Processing In this paper, we take a step towards jointly modeling automatic speech recognition (STT) and speech synthesis (TTS) in a fully non-autoregressive way. We develop a novel multimodal framework capable of handling the speech and text modalities as input either individually or together. The proposed model can also be trained with unpaired speech or text data owing to its multimodal nature. We further propose an iterative refinement strategy to improve the STT and TTS performance of our model such that the partial hypothesis at the output can be fed back to the input of our model, thus iteratively improving both STT and TTS predictions. We show that our joint model can effectively perform both STT and TTS tasks, outperforming the STT-specific baseline in all tasks and performing competitively with the TTS-specific baseline across a wide range of evaluation metrics. |
| title | A Non-autoregressive Model for Joint STT and TTS |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.09104 |