SimulTron: On-Device Simultaneous Speech to Speech Translation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913376939016192 |
|---|---|
| author | Agranovich, Alex Nachmani, Eliya Rybakov, Oleg Ding, Yifan Jia, Ye Bar, Nadav Zen, Heiga Ramanovich, Michelle Tadmor |
| author_facet | Agranovich, Alex Nachmani, Eliya Rybakov, Oleg Ding, Yifan Jia, Ye Bar, Nadav Zen, Heiga Ramanovich, Michelle Tadmor |
| contents | Simultaneous speech-to-speech translation (S2ST) holds the promise of breaking down communication barriers and enabling fluid conversations across languages. However, achieving accurate, real-time translation through mobile devices remains a major challenge. We introduce SimulTron, a novel S2ST architecture designed to tackle this task. SimulTron is a lightweight direct S2ST model that uses the strengths of the Translatotron framework while incorporating key modifications for streaming operation, and an adjustable fixed delay. Our experiments show that SimulTron surpasses Translatotron 2 in offline evaluations. Furthermore, real-time evaluations reveal that SimulTron improves upon the performance achieved by Translatotron 1. Additionally, SimulTron achieves superior BLEU scores and latency compared to previous real-time S2ST method on the MuST-C dataset. Significantly, we have successfully deployed SimulTron on a Pixel 7 Pro device, show its potential for simultaneous S2ST on-device. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02133 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SimulTron: On-Device Simultaneous Speech to Speech Translation Agranovich, Alex Nachmani, Eliya Rybakov, Oleg Ding, Yifan Jia, Ye Bar, Nadav Zen, Heiga Ramanovich, Michelle Tadmor Audio and Speech Processing Computation and Language Machine Learning Sound Simultaneous speech-to-speech translation (S2ST) holds the promise of breaking down communication barriers and enabling fluid conversations across languages. However, achieving accurate, real-time translation through mobile devices remains a major challenge. We introduce SimulTron, a novel S2ST architecture designed to tackle this task. SimulTron is a lightweight direct S2ST model that uses the strengths of the Translatotron framework while incorporating key modifications for streaming operation, and an adjustable fixed delay. Our experiments show that SimulTron surpasses Translatotron 2 in offline evaluations. Furthermore, real-time evaluations reveal that SimulTron improves upon the performance achieved by Translatotron 1. Additionally, SimulTron achieves superior BLEU scores and latency compared to previous real-time S2ST method on the MuST-C dataset. Significantly, we have successfully deployed SimulTron on a Pixel 7 Pro device, show its potential for simultaneous S2ST on-device. |
| title | SimulTron: On-Device Simultaneous Speech to Speech Translation |
| topic | Audio and Speech Processing Computation and Language Machine Learning Sound |
| url | https://arxiv.org/abs/2406.02133 |