Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.12197 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911060514045952 |
|---|---|
| author | Han, Yichen Hao, Xiaoyang Chen, Keming Xiong, Weibo He, Jun Zhang, Ruonan Cao, Junjie Liu, Yue Li, Bowen Zhang, Dongrui Xia, Hui Fu, Huilei Jia, Kai Guo, Kaixuan Jin, Mingli Meng, Qingyun Ma, Ruidong Fang, Ruiqian Guo, Shaotong Li, Xuhui Xiang, Yang Zhang, Ying Liu, Yulong Li, Yunfeng Zhang, Yuyi Zhou, Yuze Wang, Zhen Chen, Zhaowen |
| author_facet | Han, Yichen Hao, Xiaoyang Chen, Keming Xiong, Weibo He, Jun Zhang, Ruonan Cao, Junjie Liu, Yue Li, Bowen Zhang, Dongrui Xia, Hui Fu, Huilei Jia, Kai Guo, Kaixuan Jin, Mingli Meng, Qingyun Ma, Ruidong Fang, Ruiqian Guo, Shaotong Li, Xuhui Xiang, Yang Zhang, Ying Liu, Yulong Li, Yunfeng Zhang, Yuyi Zhou, Yuze Wang, Zhen Chen, Zhaowen |
| contents | Text-to-speech (TTS) synthesis has seen renewed progress under the discrete modeling paradigm. Existing autoregressive approaches often rely on single-codebook representations, which suffer from significant information loss. Even with post-hoc refinement techniques such as flow matching, these methods fail to recover fine-grained details (e.g., prosodic nuances, speaker-specific timbres), especially in challenging scenarios like singing voice or music synthesis. We propose QTTS, a novel TTS framework built upon our new audio codec, QDAC. The core innovation of QDAC lies in its end-to-end training of an ASR-based auto-regressive network with a GAN, which achieves superior semantic feature disentanglement for scalable, near-lossless compression. QTTS models these discrete codes using two innovative strategies: the Hierarchical Parallel architecture, which uses a dual-AR structure to model inter-codebook dependencies for higher-quality synthesis, and the Delay Multihead approach, which employs parallelized prediction with a fixed delay to accelerate inference speed. Our experiments demonstrate that the proposed framework achieves higher synthesis quality and better preserves expressive content compared to baseline. This suggests that scaling up compression via multi-codebook modeling is a promising direction for high-fidelity, general-purpose speech and audio generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_12197 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representations Han, Yichen Hao, Xiaoyang Chen, Keming Xiong, Weibo He, Jun Zhang, Ruonan Cao, Junjie Liu, Yue Li, Bowen Zhang, Dongrui Xia, Hui Fu, Huilei Jia, Kai Guo, Kaixuan Jin, Mingli Meng, Qingyun Ma, Ruidong Fang, Ruiqian Guo, Shaotong Li, Xuhui Xiang, Yang Zhang, Ying Liu, Yulong Li, Yunfeng Zhang, Yuyi Zhou, Yuze Wang, Zhen Chen, Zhaowen Sound Artificial Intelligence Text-to-speech (TTS) synthesis has seen renewed progress under the discrete modeling paradigm. Existing autoregressive approaches often rely on single-codebook representations, which suffer from significant information loss. Even with post-hoc refinement techniques such as flow matching, these methods fail to recover fine-grained details (e.g., prosodic nuances, speaker-specific timbres), especially in challenging scenarios like singing voice or music synthesis. We propose QTTS, a novel TTS framework built upon our new audio codec, QDAC. The core innovation of QDAC lies in its end-to-end training of an ASR-based auto-regressive network with a GAN, which achieves superior semantic feature disentanglement for scalable, near-lossless compression. QTTS models these discrete codes using two innovative strategies: the Hierarchical Parallel architecture, which uses a dual-AR structure to model inter-codebook dependencies for higher-quality synthesis, and the Delay Multihead approach, which employs parallelized prediction with a fixed delay to accelerate inference speed. Our experiments demonstrate that the proposed framework achieves higher synthesis quality and better preserves expressive content compared to baseline. This suggests that scaling up compression via multi-codebook modeling is a promising direction for high-fidelity, general-purpose speech and audio generation. |
| title | Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representations |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2507.12197 |