NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910975343460352 |
|---|---|
| author | Hasan, Md. Arid Hasanain, Maram Ahmad, Fatema Laskar, Sahinur Rahman Upadhyay, Sunaya Sukhadia, Vrunda N Kutlu, Mucahid Chowdhury, Shammur Absar Alam, Firoj |
| author_facet | Hasan, Md. Arid Hasanain, Maram Ahmad, Fatema Laskar, Sahinur Rahman Upadhyay, Sunaya Sukhadia, Vrunda N Kutlu, Mucahid Chowdhury, Shammur Absar Alam, Firoj |
| contents | Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work has been done in parallel, there is a notable lack of a framework and large scale region-specific datasets queried by native users in their own languages. This gap hinders the effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of ~64k manually annotated QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark open- and closed-source LLMs with the MultiNativQA dataset. We made the MultiNativQA dataset(https://huggingface.co/datasets/QCRI/MultiNativQA), and other experimental scripts(https://gitlab.com/nativqa/multinativqa) publicly available for the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09823 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NativQA: Multilingual Culturally-Aligned Natural Query for LLMs Hasan, Md. Arid Hasanain, Maram Ahmad, Fatema Laskar, Sahinur Rahman Upadhyay, Sunaya Sukhadia, Vrunda N Kutlu, Mucahid Chowdhury, Shammur Absar Alam, Firoj Computation and Language Artificial Intelligence 68T50 F.2.2; I.2.7 Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work has been done in parallel, there is a notable lack of a framework and large scale region-specific datasets queried by native users in their own languages. This gap hinders the effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of ~64k manually annotated QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark open- and closed-source LLMs with the MultiNativQA dataset. We made the MultiNativQA dataset(https://huggingface.co/datasets/QCRI/MultiNativQA), and other experimental scripts(https://gitlab.com/nativqa/multinativqa) publicly available for the community. |
| title | NativQA: Multilingual Culturally-Aligned Natural Query for LLMs |
| topic | Computation and Language Artificial Intelligence 68T50 F.2.2; I.2.7 |
| url | https://arxiv.org/abs/2407.09823 |