Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.00273 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911744443547648 |
|---|---|
| author | Yang, Chih-Kai Huang, Kuan-Po Lu, Ke-Han Kuan, Chun-Yi Hsiao, Chi-Yuan Lee, Hung-yi |
| author_facet | Yang, Chih-Kai Huang, Kuan-Po Lu, Ke-Han Kuan, Chun-Yi Hsiao, Chi-Yuan Lee, Hung-yi |
| contents | This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00273 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision Yang, Chih-Kai Huang, Kuan-Po Lu, Ke-Han Kuan, Chun-Yi Hsiao, Chi-Yuan Lee, Hung-yi Audio and Speech Processing Computation and Language This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks. |
| title | Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2401.00273 |