Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.10163 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911404718555136 |
|---|---|
| author | Hou, Yutao Luo, Yajing Ruan, Zhiwen Wang, Hongru Ge, Weifeng Chen, Yun Chen, Guanhua |
| author_facet | Hou, Yutao Luo, Yajing Ruan, Zhiwen Wang, Hongru Ge, Weifeng Chen, Yun Chen, Guanhua |
| contents | Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmarks. However, most benchmarks typically measure the ability of LLMs to respond to individual questions, neglecting the complex interactions in real-world applications. We introduce Compound Question Synthesis (CQ-Syn) to build Compound-QA, a benchmark targeting questions composed of multiple interrelated sub-questions. This benchmark is derived from existing QA datasets, annotated with proprietary LLMs, and verified by humans for accuracy. It encompasses five categories: Factual-Statement, Cause-and-Effect, Hypothetical-Analysis, Comparison-and-Selection, and Evaluation-and-Suggestion. It evaluates the LLM capability in terms of three dimensions, including understanding, reasoning, and knowledge. Evaluating nine open-source LLMs on Compound-QA reveals that their performance on compound questions is notably lower than on non-compound questions. We further explore strategies to enhance LLMs' handling of compound questions, and our results show that these methods substantially improve models' comprehension and reasoning abilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10163 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions Hou, Yutao Luo, Yajing Ruan, Zhiwen Wang, Hongru Ge, Weifeng Chen, Yun Chen, Guanhua Computation and Language Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmarks. However, most benchmarks typically measure the ability of LLMs to respond to individual questions, neglecting the complex interactions in real-world applications. We introduce Compound Question Synthesis (CQ-Syn) to build Compound-QA, a benchmark targeting questions composed of multiple interrelated sub-questions. This benchmark is derived from existing QA datasets, annotated with proprietary LLMs, and verified by humans for accuracy. It encompasses five categories: Factual-Statement, Cause-and-Effect, Hypothetical-Analysis, Comparison-and-Selection, and Evaluation-and-Suggestion. It evaluates the LLM capability in terms of three dimensions, including understanding, reasoning, and knowledge. Evaluating nine open-source LLMs on Compound-QA reveals that their performance on compound questions is notably lower than on non-compound questions. We further explore strategies to enhance LLMs' handling of compound questions, and our results show that these methods substantially improve models' comprehension and reasoning abilities. |
| title | Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.10163 |