Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909099122229248 |
|---|---|
| author | Fang, Hung-Chieh Ye, Nai-Xuan Shih, Yi-Jen Peng, Puyuan Wang, Hsuan-Fu Berry, Layne Lee, Hung-yi Harwath, David |
| author_facet | Fang, Hung-Chieh Ye, Nai-Xuan Shih, Yi-Jen Peng, Puyuan Wang, Hsuan-Fu Berry, Layne Lee, Hung-yi Harwath, David |
| contents | Recent advances in self-supervised speech models have shown significant improvement in many downstream tasks. However, these models predominantly centered on frame-level training objectives, which can fall short in spoken language understanding tasks that require semantic comprehension. Existing works often rely on additional speech-text data as intermediate targets, which is costly in the real-world setting. To address this challenge, we propose Pseudo-Word HuBERT (PW-HuBERT), a framework that integrates pseudo word-level targets into the training process, where the targets are derived from a visually-ground speech model, notably eliminating the need for speech-text paired data. Our experimental results on four spoken language understanding (SLU) benchmarks suggest the superiority of our model in capturing semantic information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05819 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model Fang, Hung-Chieh Ye, Nai-Xuan Shih, Yi-Jen Peng, Puyuan Wang, Hsuan-Fu Berry, Layne Lee, Hung-yi Harwath, David Audio and Speech Processing Computation and Language Machine Learning Recent advances in self-supervised speech models have shown significant improvement in many downstream tasks. However, these models predominantly centered on frame-level training objectives, which can fall short in spoken language understanding tasks that require semantic comprehension. Existing works often rely on additional speech-text data as intermediate targets, which is costly in the real-world setting. To address this challenge, we propose Pseudo-Word HuBERT (PW-HuBERT), a framework that integrates pseudo word-level targets into the training process, where the targets are derived from a visually-ground speech model, notably eliminating the need for speech-text paired data. Our experimental results on four spoken language understanding (SLU) benchmarks suggest the superiority of our model in capturing semantic information. |
| title | Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model |
| topic | Audio and Speech Processing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.05819 |