SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914674119802880 |
|---|---|
| author | Wang, Hsuan-Fu Shih, Yi-Jen Chang, Heng-Jui Berry, Layne Peng, Puyuan Lee, Hung-yi Wang, Hsin-Min Harwath, David |
| author_facet | Wang, Hsuan-Fu Shih, Yi-Jen Chang, Heng-Jui Berry, Layne Peng, Puyuan Lee, Hung-yi Wang, Hsin-Min Harwath, David |
| contents | The recently proposed visually grounded speech model SpeechCLIP is an innovative framework that bridges speech and text through images via CLIP without relying on text transcription. On this basis, this paper introduces two extensions to SpeechCLIP. First, we apply the Continuous Integrate-and-Fire (CIF) module to replace a fixed number of CLS tokens in the cascaded architecture. Second, we propose a new hybrid architecture that merges the cascaded and parallel architectures of SpeechCLIP into a multi-task learning framework. Our experimental evaluation is performed on the Flickr8k and SpokenCOCO datasets. The results show that in the speech keyword extraction task, the CIF-based cascaded SpeechCLIP model outperforms the previous cascaded SpeechCLIP model using a fixed number of CLS tokens. Furthermore, through our hybrid architecture, cascaded task learning boosts the performance of the parallel branch in image-speech retrieval tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_06959 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data Wang, Hsuan-Fu Shih, Yi-Jen Chang, Heng-Jui Berry, Layne Peng, Puyuan Lee, Hung-yi Wang, Hsin-Min Harwath, David Computation and Language Sound Audio and Speech Processing The recently proposed visually grounded speech model SpeechCLIP is an innovative framework that bridges speech and text through images via CLIP without relying on text transcription. On this basis, this paper introduces two extensions to SpeechCLIP. First, we apply the Continuous Integrate-and-Fire (CIF) module to replace a fixed number of CLS tokens in the cascaded architecture. Second, we propose a new hybrid architecture that merges the cascaded and parallel architectures of SpeechCLIP into a multi-task learning framework. Our experimental evaluation is performed on the Flickr8k and SpokenCOCO datasets. The results show that in the speech keyword extraction task, the CIF-based cascaded SpeechCLIP model outperforms the previous cascaded SpeechCLIP model using a fixed number of CLS tokens. Furthermore, through our hybrid architecture, cascaded task learning boosts the performance of the parallel branch in image-speech retrieval tasks. |
| title | SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2402.06959 |