Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.21968 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916971506827264 |
|---|---|
| author | Chen, Yushen Hu, Kai Zhou, Long Feng, Shulin Yang, Xusheng Chen, Hangting Chen, Xie |
| author_facet | Chen, Yushen Hu, Kai Zhou, Long Feng, Shulin Yang, Xusheng Chen, Hangting Chen, Xie |
| contents | We propose AUV, a unified neural audio codec with a single codebook, which enables a favourable reconstruction of speech and further extends to general audio, including vocal, music, and sound. AUV is capable of tackling any 16 kHz mixed-domain audio segment at bit rates around 700 bps. To accomplish this, we guide the matryoshka codebook with nested domain-specific partitions, assigned with corresponding teacher models to perform distillation, all in a single-stage training. A conformer-style encoder-decoder architecture with STFT features as audio representation is employed, yielding better audio quality. Comprehensive evaluations demonstrate that AUV exhibits comparable audio reconstruction ability to state-of-the-art domain-specific single-layer quantizer codecs, showcasing the potential of audio universal vector quantization with a single codebook. The pre-trained model and demo samples are available at https://swivid.github.io/AUV/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21968 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook Chen, Yushen Hu, Kai Zhou, Long Feng, Shulin Yang, Xusheng Chen, Hangting Chen, Xie Audio and Speech Processing Sound We propose AUV, a unified neural audio codec with a single codebook, which enables a favourable reconstruction of speech and further extends to general audio, including vocal, music, and sound. AUV is capable of tackling any 16 kHz mixed-domain audio segment at bit rates around 700 bps. To accomplish this, we guide the matryoshka codebook with nested domain-specific partitions, assigned with corresponding teacher models to perform distillation, all in a single-stage training. A conformer-style encoder-decoder architecture with STFT features as audio representation is employed, yielding better audio quality. Comprehensive evaluations demonstrate that AUV exhibits comparable audio reconstruction ability to state-of-the-art domain-specific single-layer quantizer codecs, showcasing the potential of audio universal vector quantization with a single codebook. The pre-trained model and demo samples are available at https://swivid.github.io/AUV/. |
| title | AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2509.21968 |