Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.13306 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915155176062976 |
|---|---|
| author | Geng, Xuelong Wei, Kun Shao, Qijie Liu, Shuiyun Lin, Zhennan Zhao, Zhixian Li, Guojian Tian, Wenjie Chen, Peikun Li, Yangze Guo, Pengcheng Shao, Mingchen Wang, Shuiyuan Cao, Yuang Wang, Chengyou Xu, Tianyi Dai, Yuhang Zhu, Xinfa Li, Yue Zhang, Li Xie, Lei |
| author_facet | Geng, Xuelong Wei, Kun Shao, Qijie Liu, Shuiyun Lin, Zhennan Zhao, Zhixian Li, Guojian Tian, Wenjie Chen, Peikun Li, Yangze Guo, Pengcheng Shao, Mingchen Wang, Shuiyuan Cao, Yuang Wang, Chengyou Xu, Tianyi Dai, Yuhang Zhu, Xinfa Li, Yue Zhang, Li Xie, Lei |
| contents | Large Language Models (LLMs) have made significant progress in various downstream tasks, inspiring the development of Speech Understanding Language Models (SULMs) to enable comprehensive speech-based interactions. However, most advanced SULMs are developed by the industry, leveraging large-scale datasets and computational resources that are not readily available to the academic community. Moreover, the lack of transparency in training details creates additional barriers to further innovation. In this study, we present OSUM, an Open Speech Understanding Model designed to explore the potential of training SLUMs under constrained academic resources. The OSUM model combines a Whisper encoder with a Qwen2 LLM and supports a wide range of speech tasks, including speech recognition (ASR), speech recognition with timestamps (SRWT), vocal event detection (VED), speech emotion recognition (SER), speaking style recognition (SSR), speaker gender classification (SGC), speaker age prediction (SAP), and speech-to-text chat (STTC). By employing an ASR+X training strategy, OSUM achieves efficient and stable multi-task training by simultaneously optimizing ASR alongside target tasks. Beyond delivering strong performance, OSUM emphasizes transparency by providing openly available data preparation and training methodologies, offering valuable insights and practical guidance for the academic community. By doing so, we aim to accelerate research and innovation in advanced SULM technologies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_13306 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia Geng, Xuelong Wei, Kun Shao, Qijie Liu, Shuiyun Lin, Zhennan Zhao, Zhixian Li, Guojian Tian, Wenjie Chen, Peikun Li, Yangze Guo, Pengcheng Shao, Mingchen Wang, Shuiyuan Cao, Yuang Wang, Chengyou Xu, Tianyi Dai, Yuhang Zhu, Xinfa Li, Yue Zhang, Li Xie, Lei Sound Computation and Language Audio and Speech Processing Large Language Models (LLMs) have made significant progress in various downstream tasks, inspiring the development of Speech Understanding Language Models (SULMs) to enable comprehensive speech-based interactions. However, most advanced SULMs are developed by the industry, leveraging large-scale datasets and computational resources that are not readily available to the academic community. Moreover, the lack of transparency in training details creates additional barriers to further innovation. In this study, we present OSUM, an Open Speech Understanding Model designed to explore the potential of training SLUMs under constrained academic resources. The OSUM model combines a Whisper encoder with a Qwen2 LLM and supports a wide range of speech tasks, including speech recognition (ASR), speech recognition with timestamps (SRWT), vocal event detection (VED), speech emotion recognition (SER), speaking style recognition (SSR), speaker gender classification (SGC), speaker age prediction (SAP), and speech-to-text chat (STTC). By employing an ASR+X training strategy, OSUM achieves efficient and stable multi-task training by simultaneously optimizing ASR alongside target tasks. Beyond delivering strong performance, OSUM emphasizes transparency by providing openly available data preparation and training methodologies, offering valuable insights and practical guidance for the academic community. By doing so, we aim to accelerate research and innovation in advanced SULM technologies. |
| title | OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia |
| topic | Sound Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.13306 |