MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911825144053760 |
|---|---|
| author | Ahuja, Sanchit Aggarwal, Divyanshu Gumma, Varun Watts, Ishaan Sathe, Ashutosh Ochieng, Millicent Hada, Rishav Jain, Prachi Axmed, Maxamed Bali, Kalika Sitaram, Sunayana |
| author_facet | Ahuja, Sanchit Aggarwal, Divyanshu Gumma, Varun Watts, Ishaan Sathe, Ashutosh Ochieng, Millicent Hada, Rishav Jain, Prachi Axmed, Maxamed Bali, Kalika Sitaram, Sunayana |
| contents | There has been a surge in LLM evaluation research to understand LLM capabilities and limitations. However, much of this research has been confined to English, leaving LLM building and evaluation for non-English languages relatively unexplored. Several new LLMs have been introduced recently, necessitating their evaluation on non-English languages. This study aims to perform a thorough evaluation of the non-English capabilities of SoTA LLMs (GPT-3.5-Turbo, GPT-4, PaLM2, Gemini-Pro, Mistral, Llama2, and Gemma) by comparing them on the same set of multilingual datasets. Our benchmark comprises 22 datasets covering 83 languages, including low-resource African languages. We also include two multimodal datasets in the benchmark and compare the performance of LLaVA models, GPT-4-Vision and Gemini-Pro-Vision. Our experiments show that larger models such as GPT-4, Gemini-Pro and PaLM2 outperform smaller models on various tasks, notably on low-resource languages, with GPT-4 outperforming PaLM2 and Gemini-Pro on more datasets. We also perform a study on data contamination and find that several models are likely to be contaminated with multilingual evaluation benchmarks, necessitating approaches to detect and handle contamination while assessing the multilingual performance of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_07463 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks Ahuja, Sanchit Aggarwal, Divyanshu Gumma, Varun Watts, Ishaan Sathe, Ashutosh Ochieng, Millicent Hada, Rishav Jain, Prachi Axmed, Maxamed Bali, Kalika Sitaram, Sunayana Computation and Language There has been a surge in LLM evaluation research to understand LLM capabilities and limitations. However, much of this research has been confined to English, leaving LLM building and evaluation for non-English languages relatively unexplored. Several new LLMs have been introduced recently, necessitating their evaluation on non-English languages. This study aims to perform a thorough evaluation of the non-English capabilities of SoTA LLMs (GPT-3.5-Turbo, GPT-4, PaLM2, Gemini-Pro, Mistral, Llama2, and Gemma) by comparing them on the same set of multilingual datasets. Our benchmark comprises 22 datasets covering 83 languages, including low-resource African languages. We also include two multimodal datasets in the benchmark and compare the performance of LLaVA models, GPT-4-Vision and Gemini-Pro-Vision. Our experiments show that larger models such as GPT-4, Gemini-Pro and PaLM2 outperform smaller models on various tasks, notably on low-resource languages, with GPT-4 outperforming PaLM2 and Gemini-Pro on more datasets. We also perform a study on data contamination and find that several models are likely to be contaminated with multilingual evaluation benchmarks, necessitating approaches to detect and handle contamination while assessing the multilingual performance of LLMs. |
| title | MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.07463 |