Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2411.09116 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916736337444864 |
|---|---|
| author | Zhang, Yidan Wan, Yu Deng, Boyi Yang, Baosong Wei, Haoran Huang, Fei Yu, Bowen Lin, Junyang Huang, Fei Zhou, Jingren |
| author_facet | Zhang, Yidan Wan, Yu Deng, Boyi Yang, Baosong Wei, Haoran Huang, Fei Yu, Bowen Lin, Junyang Huang, Fei Zhou, Jingren |
| contents | Recent advancements in large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning. Previous assessments often limited their scope to fundamental natural language processing (NLP) or isolated capability-specific tasks. To alleviate this drawback, we aim to present a comprehensive multilingual multitask benchmark. First, we introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets. Furthermore, P-MMEval delivers consistent language coverage across various datasets and provides parallel samples. Finally, we conduct extensive experiments on representative multilingual model series to compare performances across models and tasks, explore the relationship between multilingual performances and factors such as tasks, model sizes, languages, and prompts, and examine the effectiveness of knowledge transfer from English to other languages. The resulting insights are intended to offer valuable guidance for future research. The dataset is available at https://huggingface.co/datasets/Qwen/P-MMEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_09116 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs Zhang, Yidan Wan, Yu Deng, Boyi Yang, Baosong Wei, Haoran Huang, Fei Yu, Bowen Lin, Junyang Huang, Fei Zhou, Jingren Computation and Language Recent advancements in large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning. Previous assessments often limited their scope to fundamental natural language processing (NLP) or isolated capability-specific tasks. To alleviate this drawback, we aim to present a comprehensive multilingual multitask benchmark. First, we introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets. Furthermore, P-MMEval delivers consistent language coverage across various datasets and provides parallel samples. Finally, we conduct extensive experiments on representative multilingual model series to compare performances across models and tasks, explore the relationship between multilingual performances and factors such as tasks, model sizes, languages, and prompts, and examine the effectiveness of knowledge transfer from English to other languages. The resulting insights are intended to offer valuable guidance for future research. The dataset is available at https://huggingface.co/datasets/Qwen/P-MMEval. |
| title | P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.09116 |