ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913704156594176 |
|---|---|
| author | Shi, Jiatong Tian, Jinchuan Wu, Yihan Jung, Jee-weon Yip, Jia Qi Masuyama, Yoshiki Chen, William Wu, Yuning Tang, Yuxun Baali, Massa Alharhi, Dareen Zhang, Dong Deng, Ruifan Srivastava, Tejes Wu, Haibin Liu, Alexander H. Raj, Bhiksha Jin, Qin Song, Ruihua Watanabe, Shinji |
| author_facet | Shi, Jiatong Tian, Jinchuan Wu, Yihan Jung, Jee-weon Yip, Jia Qi Masuyama, Yoshiki Chen, William Wu, Yuning Tang, Yuxun Baali, Massa Alharhi, Dareen Zhang, Dong Deng, Ruifan Srivastava, Tejes Wu, Haibin Liu, Alexander H. Raj, Bhiksha Jin, Qin Song, Ruihua Watanabe, Shinji |
| contents | Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream applications are investigated, challenges have arisen in ensuring fair comparisons across diverse applications. To address these issues, we present a new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on neural codec training and evaluation. ESPnet-Codec offers various recipes in audio, music, and speech for training and evaluation using several widely adopted codec models. Together with ESPnet-Codec, we present VERSA, a standalone evaluation toolkit, which provides a comprehensive evaluation of codec performance over 20 audio evaluation metrics. Notably, we demonstrate that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15897 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech Shi, Jiatong Tian, Jinchuan Wu, Yihan Jung, Jee-weon Yip, Jia Qi Masuyama, Yoshiki Chen, William Wu, Yuning Tang, Yuxun Baali, Massa Alharhi, Dareen Zhang, Dong Deng, Ruifan Srivastava, Tejes Wu, Haibin Liu, Alexander H. Raj, Bhiksha Jin, Qin Song, Ruihua Watanabe, Shinji Audio and Speech Processing Sound Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream applications are investigated, challenges have arisen in ensuring fair comparisons across diverse applications. To address these issues, we present a new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on neural codec training and evaluation. ESPnet-Codec offers various recipes in audio, music, and speech for training and evaluation using several widely adopted codec models. Together with ESPnet-Codec, we present VERSA, a standalone evaluation toolkit, which provides a comprehensive evaluation of codec performance over 20 audio evaluation metrics. Notably, we demonstrate that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse applications. |
| title | ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2409.15897 |