Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.12383 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917529746669568 |
|---|---|
| author | Cheng, Changhao Wang, Wei Zhang, Wangyou Jia, Dongya Wu, Jian Chen, Zhuo Qian, Yanmin |
| author_facet | Cheng, Changhao Wang, Wei Zhang, Wangyou Jia, Dongya Wu, Jian Chen, Zhuo Qian, Yanmin |
| contents | Continuous speech representations based on Variational Autoencoders (VAEs) have emerged as a promising alternative to traditional spectrogram or discrete token based features for speech generation and reconstruction. Recent research has tried to enrich the structural information in VAE latent representations by aligning with self-supervised learning (SSL) features, aiming for better generation performance. However, it remains unclear whether the widely-used alignment approach based on time-axis distillation is optimal when considering more tasks. To address this problem, this paper systematically explores different alignment approaches and analyzes their impact on the performances over three axes: reconstruction, understanding, and generation. We investigate various design choices in the distillation loss. Extensive experiments show that the joint-marginal alignment approach with adaptive weighting can achieve the best overall performance while allowing for a controllable balance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12383 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation Cheng, Changhao Wang, Wei Zhang, Wangyou Jia, Dongya Wu, Jian Chen, Zhuo Qian, Yanmin Sound Continuous speech representations based on Variational Autoencoders (VAEs) have emerged as a promising alternative to traditional spectrogram or discrete token based features for speech generation and reconstruction. Recent research has tried to enrich the structural information in VAE latent representations by aligning with self-supervised learning (SSL) features, aiming for better generation performance. However, it remains unclear whether the widely-used alignment approach based on time-axis distillation is optimal when considering more tasks. To address this problem, this paper systematically explores different alignment approaches and analyzes their impact on the performances over three axes: reconstruction, understanding, and generation. We investigate various design choices in the distillation loss. Extensive experiments show that the joint-marginal alignment approach with adaptive weighting can achieve the best overall performance while allowing for a controllable balance. |
| title | On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation |
| topic | Sound |
| url | https://arxiv.org/abs/2604.12383 |