CMU's IWSLT 2024 Simultaneous Speech Translation System
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914912858537984 |
|---|---|
| author | Xu, Xi Ouyang, Siqi Yan, Brian Fernandes, Patrick Chen, William Li, Lei Neubig, Graham Watanabe, Shinji |
| author_facet | Xu, Xi Ouyang, Siqi Yan, Brian Fernandes, Patrick Chen, William Li, Lei Neubig, Graham Watanabe, Shinji |
| contents | This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07452 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CMU's IWSLT 2024 Simultaneous Speech Translation System Xu, Xi Ouyang, Siqi Yan, Brian Fernandes, Patrick Chen, William Li, Lei Neubig, Graham Watanabe, Shinji Computation and Language Artificial Intelligence This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON. |
| title | CMU's IWSLT 2024 Simultaneous Speech Translation System |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.07452 |