QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915065874087936 |
|---|---|
| author | Khan, Mohammad Aflah Yadav, Neemesh Masud, Sarah Akhtar, Md. Shad |
| author_facet | Khan, Mohammad Aflah Yadav, Neemesh Masud, Sarah Akhtar, Md. Shad |
| contents | The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated and transcribed from YouTube quiz videos. QUENCH possesses masked entities and rationales for the LLMs to predict via generation. At the intersection of geographical context and common sense reasoning, QUENCH helps assess world knowledge and deduction capabilities of LLMs via a zero-shot, open-domain quizzing setup. We perform an extensive evaluation on 7 LLMs and 4 metrics, investigating the influence of model size, prompting style, geographical context, and gold-labeled rationale generation. The benchmarking concludes with an error analysis to which the LLMs are prone. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11763 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs Khan, Mohammad Aflah Yadav, Neemesh Masud, Sarah Akhtar, Md. Shad Computation and Language The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated and transcribed from YouTube quiz videos. QUENCH possesses masked entities and rationales for the LLMs to predict via generation. At the intersection of geographical context and common sense reasoning, QUENCH helps assess world knowledge and deduction capabilities of LLMs via a zero-shot, open-domain quizzing setup. We perform an extensive evaluation on 7 LLMs and 4 metrics, investigating the influence of model size, prompting style, geographical context, and gold-labeled rationale generation. The benchmarking concludes with an error analysis to which the LLMs are prone. |
| title | QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.11763 |