SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911472198615040 |
|---|---|
| author | Xie, Zeyu Li, Chenxing Jin, Qiao Xu, Xuenan Yang, Guanrou Wang, Wenfu Wu, Mengyue Yu, Dong Zou, Yuexian |
| author_facet | Xie, Zeyu Li, Chenxing Jin, Qiao Xu, Xuenan Yang, Guanrou Wang, Wenfu Wu, Mengyue Yu, Dong Zou, Yuexian |
| contents | Recent audio generation models typically rely on Variational Autoencoders (VAEs) and perform generation within the VAE latent space. Although VAEs excel at compression and reconstruction, their latents inherently encode low-level acoustic details rather than semantically discriminative information, leading to entangled event semantics and complicating the training of generative models. To address these issues, we discard VAE acoustic latents and introduce semantic encoder latents, thereby proposing SemanticVocoder, a generative vocoder that directly synthesizes waveforms from semantic latents. Equipped with SemanticVocoder, our text-to-audio generation model achieves a Frechet Distance of 12.823 and a Frechet Audio Distance of 1.709 on the AudioCaps test set, as the introduced semantic latents exhibit superior discriminability compared to acoustic VAE latents. Beyond improved generation performance, it also serves as a promising attempt towards unifying audio understanding and generation within a shared semantic space. Generated samples are available at https://zeyuxie29.github.io/SemanticVocoder/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23333 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents Xie, Zeyu Li, Chenxing Jin, Qiao Xu, Xuenan Yang, Guanrou Wang, Wenfu Wu, Mengyue Yu, Dong Zou, Yuexian Sound 68Txx I.2 Recent audio generation models typically rely on Variational Autoencoders (VAEs) and perform generation within the VAE latent space. Although VAEs excel at compression and reconstruction, their latents inherently encode low-level acoustic details rather than semantically discriminative information, leading to entangled event semantics and complicating the training of generative models. To address these issues, we discard VAE acoustic latents and introduce semantic encoder latents, thereby proposing SemanticVocoder, a generative vocoder that directly synthesizes waveforms from semantic latents. Equipped with SemanticVocoder, our text-to-audio generation model achieves a Frechet Distance of 12.823 and a Frechet Audio Distance of 1.709 on the AudioCaps test set, as the introduced semantic latents exhibit superior discriminability compared to acoustic VAE latents. Beyond improved generation performance, it also serves as a promising attempt towards unifying audio understanding and generation within a shared semantic space. Generated samples are available at https://zeyuxie29.github.io/SemanticVocoder/. |
| title | SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents |
| topic | Sound 68Txx I.2 |
| url | https://arxiv.org/abs/2602.23333 |