IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915125359804416 |
|---|---|
| author | KJ, Sankalp Kumar, Ashutosh Balaji, Laxmaan Kotecha, Nikunj Jain, Vinija Chadha, Aman Bhaduri, Sreyoshi |
| author_facet | KJ, Sankalp Kumar, Ashutosh Balaji, Laxmaan Kotecha, Nikunj Jain, Vinija Chadha, Aman Bhaduri, Sreyoshi |
| contents | Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15747 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding KJ, Sankalp Kumar, Ashutosh Balaji, Laxmaan Kotecha, Nikunj Jain, Vinija Chadha, Aman Bhaduri, Sreyoshi Computation and Language Artificial Intelligence Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models. |
| title | IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.15747 |