Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.21877 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911586133737472 |
|---|---|
| author | Agarwal, Parth Kommuri, Navya Garg, Trizal Singhal, Prisha Shah, Dhruv Devraj, Vaibhav Sinha, Yash Challa, Jagat Sesh Mandal, Murari Kumar, Dhruv |
| author_facet | Agarwal, Parth Kommuri, Navya Garg, Trizal Singhal, Prisha Shah, Dhruv Devraj, Vaibhav Sinha, Yash Challa, Jagat Sesh Mandal, Murari Kumar, Dhruv |
| contents | Cricket is the second most popular sport worldwide, with billions of fans seeking advanced statistical insights unavailable through standard web searches. Although LLMs have advanced significantly in Text-to-SQL tasks, their capability to handle domain-specific nuances and multilingual requirements in sports analytics remains under-explored. We present CricBench, a benchmark suite evaluating the intrinsic SQL generation abilities of LLMs on cricket data across four formats: Test, ODI, T20I, and IPL. We curate a Gold-Standard dataset of 2,654 evaluation instances across four languages (English, Hindi, Punjabi, and Telugu). We evaluate seven models, GPT-5 Mini, Claude Sonnet 4, DeepSeek R1 and V3, Qwen 235B, Llama 3.1, and Gemma 2, using schema-only prompting. No single model dominates across all formats: GPT-5 Mini leads on Test cricket (12.4% DMA), Qwen 235B leads on IPL (28.7%) and T20I (17.5%), and all models score 0% on hard ODI queries. All models show a stark disconnect between syntactic validity (>98% execution accuracy) and semantic correctness (<29% DMA), with a domain gap of 37-55 percentage points versus BIRD. To our knowledge, CricBench is the first Text-to-SQL benchmark for cricket analytics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_21877 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics Agarwal, Parth Kommuri, Navya Garg, Trizal Singhal, Prisha Shah, Dhruv Devraj, Vaibhav Sinha, Yash Challa, Jagat Sesh Mandal, Murari Kumar, Dhruv Computation and Language Artificial Intelligence Cricket is the second most popular sport worldwide, with billions of fans seeking advanced statistical insights unavailable through standard web searches. Although LLMs have advanced significantly in Text-to-SQL tasks, their capability to handle domain-specific nuances and multilingual requirements in sports analytics remains under-explored. We present CricBench, a benchmark suite evaluating the intrinsic SQL generation abilities of LLMs on cricket data across four formats: Test, ODI, T20I, and IPL. We curate a Gold-Standard dataset of 2,654 evaluation instances across four languages (English, Hindi, Punjabi, and Telugu). We evaluate seven models, GPT-5 Mini, Claude Sonnet 4, DeepSeek R1 and V3, Qwen 235B, Llama 3.1, and Gemma 2, using schema-only prompting. No single model dominates across all formats: GPT-5 Mini leads on Test cricket (12.4% DMA), Qwen 235B leads on IPL (28.7%) and T20I (17.5%), and all models score 0% on hard ODI queries. All models show a stark disconnect between syntactic validity (>98% execution accuracy) and semantic correctness (<29% DMA), with a domain gap of 37-55 percentage points versus BIRD. To our knowledge, CricBench is the first Text-to-SQL benchmark for cricket analytics. |
| title | CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.21877 |