UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913714324635648 |
|---|---|
| author | Liu, Alexander H. Lee, Sang-gil Yang, Chao-Han Huck Gong, Yuan Wang, Yu-Chiang Frank Glass, James R. Valle, Rafael Catanzaro, Bryan |
| author_facet | Liu, Alexander H. Lee, Sang-gil Yang, Chao-Han Huck Gong, Yuan Wang, Yu-Chiang Frank Glass, James R. Valle, Rafael Catanzaro, Bryan |
| contents | Pre-training and representation learning have been playing an increasingly important role in modern speech processing. Nevertheless, different applications have been relying on different foundation models, since predominant pre-training techniques are either designed for discriminative tasks or generative tasks. In this work, we make the first attempt at building a unified pre-training framework for both types of tasks in speech. We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks. We propose UniWav, an encoder-decoder framework designed to unify pre-training representation learning and generative tasks. On speech recognition, text-to-speech, and speech tokenization, UniWav achieves comparable performance to different existing foundation models, each trained on a specific task. Our findings suggest that a single general-purpose foundation model for speech can be built to replace different foundation models, reducing the overhead and cost of pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_00733 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation Liu, Alexander H. Lee, Sang-gil Yang, Chao-Han Huck Gong, Yuan Wang, Yu-Chiang Frank Glass, James R. Valle, Rafael Catanzaro, Bryan Audio and Speech Processing Computation and Language Sound Pre-training and representation learning have been playing an increasingly important role in modern speech processing. Nevertheless, different applications have been relying on different foundation models, since predominant pre-training techniques are either designed for discriminative tasks or generative tasks. In this work, we make the first attempt at building a unified pre-training framework for both types of tasks in speech. We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks. We propose UniWav, an encoder-decoder framework designed to unify pre-training representation learning and generative tasks. On speech recognition, text-to-speech, and speech tokenization, UniWav achieves comparable performance to different existing foundation models, each trained on a specific task. Our findings suggest that a single general-purpose foundation model for speech can be built to replace different foundation models, reducing the overhead and cost of pre-training. |
| title | UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2503.00733 |