Language translation, and change of accent for speech-to-speech task using diffusion model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909604361797632 |
|---|---|
| author | Mishra, Abhishek Chowdhury, Ritesh Sur Bahuguna, Vartul Pandey, Isha Ramakrishnan, Ganesh |
| author_facet | Mishra, Abhishek Chowdhury, Ritesh Sur Bahuguna, Vartul Pandey, Isha Ramakrishnan, Ganesh |
| contents | Speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another, typically focusing on either language translation or accent adaptation. However, effective cross-cultural communication requires handling both aspects simultaneously - translating content while adapting the speaker's accent to match the target language context. In this work, we propose a unified approach for simultaneous speech translation and change of accent, a task that remains underexplored in current literature. Our method reformulates the problem as a conditional generation task, where target speech is generated based on phonemes and guided by target speech features. Leveraging the power of diffusion models, known for high-fidelity generative capabilities, we adapt text-to-image diffusion strategies by conditioning on source speech transcriptions and generating Mel spectrograms representing the target speech with desired linguistic and accentual attributes. This integrated framework enables joint optimization of translation and accent adaptation, offering a more parameter-efficient and effective model compared to traditional pipelines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_04639 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Language translation, and change of accent for speech-to-speech task using diffusion model Mishra, Abhishek Chowdhury, Ritesh Sur Bahuguna, Vartul Pandey, Isha Ramakrishnan, Ganesh Computation and Language Artificial Intelligence Speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another, typically focusing on either language translation or accent adaptation. However, effective cross-cultural communication requires handling both aspects simultaneously - translating content while adapting the speaker's accent to match the target language context. In this work, we propose a unified approach for simultaneous speech translation and change of accent, a task that remains underexplored in current literature. Our method reformulates the problem as a conditional generation task, where target speech is generated based on phonemes and guided by target speech features. Leveraging the power of diffusion models, known for high-fidelity generative capabilities, we adapt text-to-image diffusion strategies by conditioning on source speech transcriptions and generating Mel spectrograms representing the target speech with desired linguistic and accentual attributes. This integrated framework enables joint optimization of translation and accent adaptation, offering a more parameter-efficient and effective model compared to traditional pipelines. |
| title | Language translation, and change of accent for speech-to-speech task using diffusion model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.04639 |