Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.16639 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915628759121920 |
|---|---|
| author | Tseng, Wei-Cheng Harwath, David |
| author_facet | Tseng, Wei-Cheng Harwath, David |
| contents | Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16639 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs Tseng, Wei-Cheng Harwath, David Audio and Speech Processing Computation and Language Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency. |
| title | Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2511.16639 |