Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.25409 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915586391408640 |
|---|---|
| author | Devane, Vijay Nauman, Mohd Patel, Bhargav Wakchoure, Aniket Mahendra Sant, Yogeshkumar Pawar, Shyam Thakur, Viraj Godse, Ananya Patra, Sunil Maurya, Neha Racha, Suraj Singh, Nitish Kamal Nagpal, Ajay Sawarkar, Piyush Pundalik, Kundeshwar Vijayrao Saluja, Rohit Ramakrishnan, Ganesh |
| author_facet | Devane, Vijay Nauman, Mohd Patel, Bhargav Wakchoure, Aniket Mahendra Sant, Yogeshkumar Pawar, Shyam Thakur, Viraj Godse, Ananya Patra, Sunil Maurya, Neha Racha, Suraj Singh, Nitish Kamal Nagpal, Ajay Sawarkar, Piyush Pundalik, Kundeshwar Vijayrao Saluja, Rohit Ramakrishnan, Ganesh |
| contents | The rapid advancement of large language models(LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, the first domain-specific, multi-task, bilingual benchmark focusing on critical Indic knowledge systems. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It spans four major domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive dataset for evaluating large language models across India's diverse knowledge domains. It enables assessment of models' ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25409 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains Devane, Vijay Nauman, Mohd Patel, Bhargav Wakchoure, Aniket Mahendra Sant, Yogeshkumar Pawar, Shyam Thakur, Viraj Godse, Ananya Patra, Sunil Maurya, Neha Racha, Suraj Singh, Nitish Kamal Nagpal, Ajay Sawarkar, Piyush Pundalik, Kundeshwar Vijayrao Saluja, Rohit Ramakrishnan, Ganesh Computation and Language Artificial Intelligence The rapid advancement of large language models(LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, the first domain-specific, multi-task, bilingual benchmark focusing on critical Indic knowledge systems. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It spans four major domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive dataset for evaluating large language models across India's diverse knowledge domains. It enables assessment of models' ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research. |
| title | BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.25409 |