Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912385753677824 |
|---|---|
| author | Ma, Hao Chen, Rujin Zhang, Xiao-Lei Liu, Ju Li, Xuelong |
| author_facet | Ma, Hao Chen, Rujin Zhang, Xiao-Lei Liu, Ju Li, Xuelong |
| contents | Target speech extraction (TSE) isolates the speech of a specific speaker from a multi-talker overlapped speech mixture. Most existing TSE models rely on discriminative methods, typically predicting a time-frequency spectrogram mask for the target speech. However, imperfections in these masks often result in over-/under-suppression of target/non-target speech, degrading perceptual quality. Generative methods, by contrast, re-synthesize target speech based on the mixture and target speaker cues, achieving superior perceptual quality. Nevertheless, these methods often overlook speech intelligibility, leading to alterations or loss of semantic content in the re-synthesized speech. Inspired by the Whisper model's success in target speaker ASR, we propose a generative TSE framework based on the pre-trained Whisper model to address the above issues. This framework integrates semantic modeling with flow-based acoustic modeling to achieve both high intelligibility and perceptual quality. Results from multiple benchmarks demonstrate that the proposed method outperforms existing generative and discriminative baselines. We present speech samples on https://aisaka0v0.github.io/GenerativeTSE_demo/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_14477 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR Ma, Hao Chen, Rujin Zhang, Xiao-Lei Liu, Ju Li, Xuelong Audio and Speech Processing Sound Target speech extraction (TSE) isolates the speech of a specific speaker from a multi-talker overlapped speech mixture. Most existing TSE models rely on discriminative methods, typically predicting a time-frequency spectrogram mask for the target speech. However, imperfections in these masks often result in over-/under-suppression of target/non-target speech, degrading perceptual quality. Generative methods, by contrast, re-synthesize target speech based on the mixture and target speaker cues, achieving superior perceptual quality. Nevertheless, these methods often overlook speech intelligibility, leading to alterations or loss of semantic content in the re-synthesized speech. Inspired by the Whisper model's success in target speaker ASR, we propose a generative TSE framework based on the pre-trained Whisper model to address the above issues. This framework integrates semantic modeling with flow-based acoustic modeling to achieve both high intelligibility and perceptual quality. Results from multiple benchmarks demonstrate that the proposed method outperforms existing generative and discriminative baselines. We present speech samples on https://aisaka0v0.github.io/GenerativeTSE_demo/. |
| title | Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2501.14477 |