Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.25150 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912675938697216 |
|---|---|
| author | Gopal, Shreyas Anshul, Ashutosh Li, Haoyang Yeo, Yue Heng Liu, Hexin Chng, Eng Siong |
| author_facet | Gopal, Shreyas Anshul, Ashutosh Li, Haoyang Yeo, Yue Heng Liu, Hexin Chng, Eng Siong |
| contents | Discrete audio representations are gaining traction in speech modeling due to their interpretability and compatibility with large language models, but are not always optimized for noisy or real-world environments. Building on existing works that quantize Whisper embeddings for speech-to-unit modeling, we propose disentangling semantic speech content from background noise in the latent space. Our end-to-end model separates clean speech in the form of codebook tokens, while extracting interpretable noise vectors as quantization residue which are supervised via a lightweight classifier. We show that our approach improves alignment between clean/noisy speech and text, producing speech tokens that display a high degree of noiseinvariance, and improves ASR performance. Keeping Whisper frozen, we show an 82% reduction in error rate compared to Whisper, and 35% improvement over baseline methods on the VBDemand test set. Further analyses show that the learned token space generalizes well to both seen and unseen acoustic conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR Gopal, Shreyas Anshul, Ashutosh Li, Haoyang Yeo, Yue Heng Liu, Hexin Chng, Eng Siong Computation and Language Discrete audio representations are gaining traction in speech modeling due to their interpretability and compatibility with large language models, but are not always optimized for noisy or real-world environments. Building on existing works that quantize Whisper embeddings for speech-to-unit modeling, we propose disentangling semantic speech content from background noise in the latent space. Our end-to-end model separates clean speech in the form of codebook tokens, while extracting interpretable noise vectors as quantization residue which are supervised via a lightweight classifier. We show that our approach improves alignment between clean/noisy speech and text, producing speech tokens that display a high degree of noiseinvariance, and improves ASR performance. Keeping Whisper frozen, we show an 82% reduction in error rate compared to Whisper, and 35% improvement over baseline methods on the VBDemand test set. Further analyses show that the learned token space generalizes well to both seen and unseen acoustic conditions. |
| title | Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.25150 |