Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2411.05361 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912420189962240 |
|---|---|
| author | Huang, Chien-yu Chen, Wei-Chih Yang, Shu-wen Liu, Andy T. Li, Chen-An Lin, Yu-Xiang Tseng, Wei-Cheng Diwan, Anuj Shih, Yi-Jen Shi, Jiatong Chen, William Yang, Chih-Kai Ren, Wenze Chen, Xuanjun Hsiao, Chi-Yuan Peng, Puyuan Wang, Shih-Heng Kuan, Chun-Yi Lu, Ke-Han Chang, Kai-Wei Ritter-Gutierrez, Fabian Huang, Kuan-Po Arora, Siddhant Lin, You-Kuan Chuang, Ming To Yeo, Eunjung Chang, Kalvin Chien, Chung-Ming Choi, Kwanghee Wang, Jun-You Hsieh, Cheng-Hsiu Lin, Yi-Cheng Yu, Chee-En Chiu, I-Hsiang Guimarães, Heitor R. Han, Jionghao Lin, Tzu-Quan Lin, Tzu-Yuan Chang, Homu Chang, Ting-Wu Chen, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Cheng, Hsi-Chun Dhawan, Kunal Fang, Jia-Lin Fang, Shi-Xin Chiang, Kuan-Yu Fang Fu, Chi An Hsiao, Hsien-Fu Hsu, Ching Yu Huang, Shao-Syuan Wei, Lee Chen Lin, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Liu, Ting-Chun Lu, Li-Chun Pai, Tsung-Min Pasad, Ankita Kuan, Shih-Yun Shan Shon, Suwon Tang, Yuxun Tsai, Yun-Shao Wei, Jui-Chiang Wei, Tzu-Chieh Wu, Chengxi Wu, Dien-Ruei Yang, Chao-Han Huck Yang, Chieh-Chi Yip, Jia Qi Yuan, Shao-Xiang Noroozi, Vahid Chen, Zhehuai Wu, Haibin Livescu, Karen Harwath, David Watanabe, Shinji Lee, Hung-yi |
| author_facet | Huang, Chien-yu Chen, Wei-Chih Yang, Shu-wen Liu, Andy T. Li, Chen-An Lin, Yu-Xiang Tseng, Wei-Cheng Diwan, Anuj Shih, Yi-Jen Shi, Jiatong Chen, William Yang, Chih-Kai Ren, Wenze Chen, Xuanjun Hsiao, Chi-Yuan Peng, Puyuan Wang, Shih-Heng Kuan, Chun-Yi Lu, Ke-Han Chang, Kai-Wei Ritter-Gutierrez, Fabian Huang, Kuan-Po Arora, Siddhant Lin, You-Kuan Chuang, Ming To Yeo, Eunjung Chang, Kalvin Chien, Chung-Ming Choi, Kwanghee Wang, Jun-You Hsieh, Cheng-Hsiu Lin, Yi-Cheng Yu, Chee-En Chiu, I-Hsiang Guimarães, Heitor R. Han, Jionghao Lin, Tzu-Quan Lin, Tzu-Yuan Chang, Homu Chang, Ting-Wu Chen, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Cheng, Hsi-Chun Dhawan, Kunal Fang, Jia-Lin Fang, Shi-Xin Chiang, Kuan-Yu Fang Fu, Chi An Hsiao, Hsien-Fu Hsu, Ching Yu Huang, Shao-Syuan Wei, Lee Chen Lin, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Liu, Ting-Chun Lu, Li-Chun Pai, Tsung-Min Pasad, Ankita Kuan, Shih-Yun Shan Shon, Suwon Tang, Yuxun Tsai, Yun-Shao Wei, Jui-Chiang Wei, Tzu-Chieh Wu, Chengxi Wu, Dien-Ruei Yang, Chao-Han Huck Yang, Chieh-Chi Yip, Jia Qi Yuan, Shao-Xiang Noroozi, Vahid Chen, Zhehuai Wu, Haibin Livescu, Karen Harwath, David Watanabe, Shinji Lee, Hung-yi |
| contents | Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results show that no model performed well universally. SALMONN-13B excelled in English ASR and Qwen2-Audio-7B-Instruct showed high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We open-source all task data and the evaluation pipeline at https://github.com/dynamic-superb/dynamic-superb. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05361 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks Huang, Chien-yu Chen, Wei-Chih Yang, Shu-wen Liu, Andy T. Li, Chen-An Lin, Yu-Xiang Tseng, Wei-Cheng Diwan, Anuj Shih, Yi-Jen Shi, Jiatong Chen, William Yang, Chih-Kai Ren, Wenze Chen, Xuanjun Hsiao, Chi-Yuan Peng, Puyuan Wang, Shih-Heng Kuan, Chun-Yi Lu, Ke-Han Chang, Kai-Wei Ritter-Gutierrez, Fabian Huang, Kuan-Po Arora, Siddhant Lin, You-Kuan Chuang, Ming To Yeo, Eunjung Chang, Kalvin Chien, Chung-Ming Choi, Kwanghee Wang, Jun-You Hsieh, Cheng-Hsiu Lin, Yi-Cheng Yu, Chee-En Chiu, I-Hsiang Guimarães, Heitor R. Han, Jionghao Lin, Tzu-Quan Lin, Tzu-Yuan Chang, Homu Chang, Ting-Wu Chen, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Cheng, Hsi-Chun Dhawan, Kunal Fang, Jia-Lin Fang, Shi-Xin Chiang, Kuan-Yu Fang Fu, Chi An Hsiao, Hsien-Fu Hsu, Ching Yu Huang, Shao-Syuan Wei, Lee Chen Lin, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Liu, Ting-Chun Lu, Li-Chun Pai, Tsung-Min Pasad, Ankita Kuan, Shih-Yun Shan Shon, Suwon Tang, Yuxun Tsai, Yun-Shao Wei, Jui-Chiang Wei, Tzu-Chieh Wu, Chengxi Wu, Dien-Ruei Yang, Chao-Han Huck Yang, Chieh-Chi Yip, Jia Qi Yuan, Shao-Xiang Noroozi, Vahid Chen, Zhehuai Wu, Haibin Livescu, Karen Harwath, David Watanabe, Shinji Lee, Hung-yi Computation and Language Audio and Speech Processing Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results show that no model performed well universally. SALMONN-13B excelled in English ASR and Qwen2-Audio-7B-Instruct showed high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We open-source all task data and the evaluation pipeline at https://github.com/dynamic-superb/dynamic-superb. |
| title | Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2411.05361 |