MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918158658437120 |
|---|---|
| author | Wang, Jianjin Zhao, Runsong Liu, Xiaoqian Ge, Yuan Xu, Ziqiang Xiao, Tong Gao, Shengxiang Yu, Zhengtao Zhu, Jingbo |
| author_facet | Wang, Jianjin Zhao, Runsong Liu, Xiaoqian Ge, Yuan Xu, Ziqiang Xiao, Tong Gao, Shengxiang Yu, Zhengtao Zhu, Jingbo |
| contents | Current direct speech-to-speech translation methods predominantly employ speech tokens as intermediate representations. However, a single speech token is not dense in semantics, so we generally need multiple tokens to express a complete semantic unit. To address this limitation, we introduce multi-token prediction (MTP) loss into speech-to-unit translation (S2UT) models, enabling models to predict multiple subsequent tokens at each position, thereby capturing more complete semantics and enhancing information density per position. Initial MTP implementations apply the loss at the final layer, which improves output representation but initiates information enrichment too late. We hypothesize that advancing the information enrichment process to intermediate layers can achieve earlier and more effective enhancement of hidden representation. Consequently, we propose MTP-S2UT loss, applying MTP loss to hidden representation where CTC loss is computed. Experiments demonstrate that all MTP loss variants consistently improve the quality of S2UT translation, with MTP-S2UT achieving the best performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10003 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Prediction Wang, Jianjin Zhao, Runsong Liu, Xiaoqian Ge, Yuan Xu, Ziqiang Xiao, Tong Gao, Shengxiang Yu, Zhengtao Zhu, Jingbo Computation and Language Sound Audio and Speech Processing Current direct speech-to-speech translation methods predominantly employ speech tokens as intermediate representations. However, a single speech token is not dense in semantics, so we generally need multiple tokens to express a complete semantic unit. To address this limitation, we introduce multi-token prediction (MTP) loss into speech-to-unit translation (S2UT) models, enabling models to predict multiple subsequent tokens at each position, thereby capturing more complete semantics and enhancing information density per position. Initial MTP implementations apply the loss at the final layer, which improves output representation but initiates information enrichment too late. We hypothesize that advancing the information enrichment process to intermediate layers can achieve earlier and more effective enhancement of hidden representation. Consequently, we propose MTP-S2UT loss, applying MTP loss to hidden representation where CTC loss is computed. Experiments demonstrate that all MTP loss variants consistently improve the quality of S2UT translation, with MTP-S2UT achieving the best performance. |
| title | MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Prediction |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2510.10003 |