Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.19799 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909408974340096 |
|---|---|
| author | Romanou, Angelika Foroutan, Negar Sotnikova, Anna Chen, Zeming Nelaturu, Sree Harsha Singh, Shivalika Maheshwary, Rishabh Altomare, Micol Haggag, Mohamed A. A, Snegha Amayuelas, Alfonso Amirudin, Azril Hafizi Aryabumi, Viraat Boiko, Danylo Chang, Michael Chim, Jenny Cohen, Gal Dalmia, Aditya Kumar Diress, Abraham Duwal, Sharad Dzenhaliou, Daniil Florez, Daniel Fernando Erazo Farestam, Fabian Imperial, Joseph Marvin Islam, Shayekh Bin Isotalo, Perttu Jabbarishiviari, Maral Karlsson, Börje F. Khalilov, Eldar Klamm, Christopher Koto, Fajri Krzemiński, Dominik de Melo, Gabriel Adriano Montariol, Syrielle Nan, Yiyang Niklaus, Joel Novikova, Jekaterina Ceron, Johan Samir Obando Paul, Debjit Ploeger, Esther Purbey, Jebish Rajwal, Swati Ravi, Selvan Sunitha Rydell, Sara Santhosh, Roshan Sharma, Drishti Skenduli, Marjana Prifti Moakhar, Arshia Soltani Moakhar, Bardia Soltani Tamir, Ran Tarun, Ayush Kumar Wasi, Azmine Toushik Weerasinghe, Thenuka Ovin Yilmaz, Serhan Zhang, Mike Schlag, Imanol Fadaee, Marzieh Hooker, Sara Bosselut, Antoine |
| author_facet | Romanou, Angelika Foroutan, Negar Sotnikova, Anna Chen, Zeming Nelaturu, Sree Harsha Singh, Shivalika Maheshwary, Rishabh Altomare, Micol Haggag, Mohamed A. A, Snegha Amayuelas, Alfonso Amirudin, Azril Hafizi Aryabumi, Viraat Boiko, Danylo Chang, Michael Chim, Jenny Cohen, Gal Dalmia, Aditya Kumar Diress, Abraham Duwal, Sharad Dzenhaliou, Daniil Florez, Daniel Fernando Erazo Farestam, Fabian Imperial, Joseph Marvin Islam, Shayekh Bin Isotalo, Perttu Jabbarishiviari, Maral Karlsson, Börje F. Khalilov, Eldar Klamm, Christopher Koto, Fajri Krzemiński, Dominik de Melo, Gabriel Adriano Montariol, Syrielle Nan, Yiyang Niklaus, Joel Novikova, Jekaterina Ceron, Johan Samir Obando Paul, Debjit Ploeger, Esther Purbey, Jebish Rajwal, Swati Ravi, Selvan Sunitha Rydell, Sara Santhosh, Roshan Sharma, Drishti Skenduli, Marjana Prifti Moakhar, Arshia Soltani Moakhar, Bardia Soltani Tamir, Ran Tarun, Ayush Kumar Wasi, Azmine Toushik Weerasinghe, Thenuka Ovin Yilmaz, Serhan Zhang, Mike Schlag, Imanol Fadaee, Marzieh Hooker, Sara Bosselut, Antoine |
| contents | The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_19799 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge Romanou, Angelika Foroutan, Negar Sotnikova, Anna Chen, Zeming Nelaturu, Sree Harsha Singh, Shivalika Maheshwary, Rishabh Altomare, Micol Haggag, Mohamed A. A, Snegha Amayuelas, Alfonso Amirudin, Azril Hafizi Aryabumi, Viraat Boiko, Danylo Chang, Michael Chim, Jenny Cohen, Gal Dalmia, Aditya Kumar Diress, Abraham Duwal, Sharad Dzenhaliou, Daniil Florez, Daniel Fernando Erazo Farestam, Fabian Imperial, Joseph Marvin Islam, Shayekh Bin Isotalo, Perttu Jabbarishiviari, Maral Karlsson, Börje F. Khalilov, Eldar Klamm, Christopher Koto, Fajri Krzemiński, Dominik de Melo, Gabriel Adriano Montariol, Syrielle Nan, Yiyang Niklaus, Joel Novikova, Jekaterina Ceron, Johan Samir Obando Paul, Debjit Ploeger, Esther Purbey, Jebish Rajwal, Swati Ravi, Selvan Sunitha Rydell, Sara Santhosh, Roshan Sharma, Drishti Skenduli, Marjana Prifti Moakhar, Arshia Soltani Moakhar, Bardia Soltani Tamir, Ran Tarun, Ayush Kumar Wasi, Azmine Toushik Weerasinghe, Thenuka Ovin Yilmaz, Serhan Zhang, Mike Schlag, Imanol Fadaee, Marzieh Hooker, Sara Bosselut, Antoine Computation and Language The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed. |
| title | INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.19799 |