PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914832347824128 |
|---|---|
| author | Yang, Runyan Yang, Huibao Zhang, Xiqing Ye, Tiantian Liu, Ying Gao, Yingying Zhang, Shilei Deng, Chao Feng, Junlan |
| author_facet | Yang, Runyan Yang, Huibao Zhang, Xiqing Ye, Tiantian Liu, Ying Gao, Yingying Zhang, Shilei Deng, Chao Feng, Junlan |
| contents | Recently, there have been attempts to integrate various speech processing tasks into a unified model. However, few previous works directly demonstrated that joint optimization of diverse tasks in multitask speech models has positive influence on the performance of individual tasks. In this paper we present a multitask speech model -- PolySpeech, which supports speech recognition, speech synthesis, and two speech classification tasks. PolySpeech takes multi-modal language model as its core structure and uses semantic representations as speech inputs. We introduce semantic speech embedding tokenization and speech reconstruction methods to PolySpeech, enabling efficient generation of high-quality speech for any given speaker. PolySpeech shows competitiveness across various tasks compared to single-task models. In our experiments, multitask optimization achieves performance comparable to single-task optimization and is especially beneficial for specific tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07801 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models Yang, Runyan Yang, Huibao Zhang, Xiqing Ye, Tiantian Liu, Ying Gao, Yingying Zhang, Shilei Deng, Chao Feng, Junlan Computation and Language Sound Audio and Speech Processing Recently, there have been attempts to integrate various speech processing tasks into a unified model. However, few previous works directly demonstrated that joint optimization of diverse tasks in multitask speech models has positive influence on the performance of individual tasks. In this paper we present a multitask speech model -- PolySpeech, which supports speech recognition, speech synthesis, and two speech classification tasks. PolySpeech takes multi-modal language model as its core structure and uses semantic representations as speech inputs. We introduce semantic speech embedding tokenization and speech reconstruction methods to PolySpeech, enabling efficient generation of high-quality speech for any given speaker. PolySpeech shows competitiveness across various tasks compared to single-task models. In our experiments, multitask optimization achieves performance comparable to single-task optimization and is especially beneficial for specific tasks. |
| title | PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.07801 |