Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913895100186624 |
|---|---|
| author | Wu, Wenxuan Wang, Shuai Wu, Xixin Meng, Helen Li, Haizhou |
| author_facet | Wu, Wenxuan Wang, Shuai Wu, Xixin Meng, Helen Li, Haizhou |
| contents | Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09792 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction Wu, Wenxuan Wang, Shuai Wu, Xixin Meng, Helen Li, Haizhou Sound Machine Learning Multimedia Audio and Speech Processing Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains. |
| title | Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction |
| topic | Sound Machine Learning Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.09792 |