Beyond Speaker Identity: Text Guided Target Speech Extraction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910786633334784 |
|---|---|
| author | Huo, Mingyue Jain, Abhinav Huynh, Cong Phuoc Kong, Fanjie Wang, Pichao Liu, Zhu Bhat, Vimal |
| author_facet | Huo, Mingyue Jain, Abhinav Huynh, Cong Phuoc Kong, Fanjie Wang, Pichao Liu, Zhu Bhat, Vimal |
| contents | Target Speech Extraction (TSE) traditionally relies on explicit clues about the speaker's identity like enrollment audio, face images, or videos, which may not always be available. In this paper, we propose a text-guided TSE model StyleTSE that uses natural language descriptions of speaking style in addition to the audio clue to extract the desired speech from a given mixture. Our model integrates a speech separation network adapted from SepFormer with a bi-modality clue network that flexibly processes both audio and text clues. To train and evaluate our model, we introduce a new dataset TextrolMix with speech mixtures and natural language descriptions. Experimental results demonstrate that our method effectively separates speech based not only on who is speaking, but also on how they are speaking, enhancing TSE in scenarios where traditional audio clues are absent. Demos are at: https://mingyue66.github.io/TextrolMix/demo/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09169 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Speaker Identity: Text Guided Target Speech Extraction Huo, Mingyue Jain, Abhinav Huynh, Cong Phuoc Kong, Fanjie Wang, Pichao Liu, Zhu Bhat, Vimal Audio and Speech Processing Sound Target Speech Extraction (TSE) traditionally relies on explicit clues about the speaker's identity like enrollment audio, face images, or videos, which may not always be available. In this paper, we propose a text-guided TSE model StyleTSE that uses natural language descriptions of speaking style in addition to the audio clue to extract the desired speech from a given mixture. Our model integrates a speech separation network adapted from SepFormer with a bi-modality clue network that flexibly processes both audio and text clues. To train and evaluate our model, we introduce a new dataset TextrolMix with speech mixtures and natural language descriptions. Experimental results demonstrate that our method effectively separates speech based not only on who is speaking, but also on how they are speaking, enhancing TSE in scenarios where traditional audio clues are absent. Demos are at: https://mingyue66.github.io/TextrolMix/demo/ |
| title | Beyond Speaker Identity: Text Guided Target Speech Extraction |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2501.09169 |