Bridging the Gap between Audio and Text using Parallel-attention for User-defined Keyword Spotting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912079961653248 |
|---|---|
| author | Kim, Youkyum Jung, Jaemin Park, Jihwan Kim, Byeong-Yeol Chung, Joon Son |
| author_facet | Kim, Youkyum Jung, Jaemin Park, Jihwan Kim, Byeong-Yeol Chung, Joon Son |
| contents | This paper proposes a novel user-defined keyword spotting framework that accurately detects audio keywords based on text enrollment. Since audio data possesses additional acoustic information compared to text, there are discrepancies between these two modalities. To address this challenge, we present ParallelKWS, which utilises self- and cross-attention in a parallel architecture to effectively capture information both within and across the two modalities. We further propose a phoneme duration-based alignment loss that enforces the sequential correspondence between audio and text features. Extensive experimental results demonstrate that our proposed method achieves state-of-the-art performance on several benchmark datasets in both seen and unseen domains, without incorporating extra data beyond the dataset used in previous studies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03593 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bridging the Gap between Audio and Text using Parallel-attention for User-defined Keyword Spotting Kim, Youkyum Jung, Jaemin Park, Jihwan Kim, Byeong-Yeol Chung, Joon Son Audio and Speech Processing This paper proposes a novel user-defined keyword spotting framework that accurately detects audio keywords based on text enrollment. Since audio data possesses additional acoustic information compared to text, there are discrepancies between these two modalities. To address this challenge, we present ParallelKWS, which utilises self- and cross-attention in a parallel architecture to effectively capture information both within and across the two modalities. We further propose a phoneme duration-based alignment loss that enforces the sequential correspondence between audio and text features. Extensive experimental results demonstrate that our proposed method achieves state-of-the-art performance on several benchmark datasets in both seen and unseen domains, without incorporating extra data beyond the dataset used in previous studies. |
| title | Bridging the Gap between Audio and Text using Parallel-attention for User-defined Keyword Spotting |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.03593 |