WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914539741642752 |
|---|---|
| author | Yang, Guanrou Tan, Tian Chen, Qian Niu, Zhikang Song, Yakun Ma, Ziyang Chen, Yushen Xie, Zeyu Wang, Tianrui Yang, Yifan Chen, Wenxi Chen, Qi Liu, Wenrui Yang, Shan Chen, Xie |
| author_facet | Yang, Guanrou Tan, Tian Chen, Qian Niu, Zhikang Song, Yakun Ma, Ziyang Chen, Yushen Xie, Zeyu Wang, Tianrui Yang, Yifan Chen, Wenxi Chen, Qi Liu, Wenrui Yang, Shan Chen, Xie |
| contents | Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06407 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling Yang, Guanrou Tan, Tian Chen, Qian Niu, Zhikang Song, Yakun Ma, Ziyang Chen, Yushen Xie, Zeyu Wang, Tianrui Yang, Yifan Chen, Wenxi Chen, Qi Liu, Wenrui Yang, Shan Chen, Xie Audio and Speech Processing Artificial Intelligence Computation and Language Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube. |
| title | WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.06407 |