Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917001305260032 |
|---|---|
| author | You, Wangjie Wang, Xusheng Wang, Xing Jiao, Wenxiang Feng, Chao Li, Juntao Zhang, Min |
| author_facet | You, Wangjie Wang, Xusheng Wang, Xing Jiao, Wenxiang Feng, Chao Li, Juntao Zhang, Min |
| contents | While Large Language Models (LLMs) have demonstrated advanced reasoning capabilities, their comprehensive evaluation in general Chinese-language contexts remains understudied. To bridge this gap, we propose Chinese Commonsense Multi-hop Reasoning (CCMOR), a novel benchmark designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning. Specifically, we first construct a domain-balanced seed set from existing QA datasets, then develop an LLM-powered pipeline to generate multi-hop questions anchored on factual unit chains. To ensure the quality of resulting dataset, we implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions. Using CCMOR, we evaluate state-of-the-art LLMs, demonstrating persistent limitations in LLMs' ability to process long-tail knowledge and execute knowledge-intensive reasoning. Notably, retrieval-augmented generation substantially mitigates these knowledge gaps, yielding significant performance gains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08800 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective You, Wangjie Wang, Xusheng Wang, Xing Jiao, Wenxiang Feng, Chao Li, Juntao Zhang, Min Computation and Language Artificial Intelligence While Large Language Models (LLMs) have demonstrated advanced reasoning capabilities, their comprehensive evaluation in general Chinese-language contexts remains understudied. To bridge this gap, we propose Chinese Commonsense Multi-hop Reasoning (CCMOR), a novel benchmark designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning. Specifically, we first construct a domain-balanced seed set from existing QA datasets, then develop an LLM-powered pipeline to generate multi-hop questions anchored on factual unit chains. To ensure the quality of resulting dataset, we implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions. Using CCMOR, we evaluate state-of-the-art LLMs, demonstrating persistent limitations in LLMs' ability to process long-tail knowledge and execute knowledge-intensive reasoning. Notably, retrieval-augmented generation substantially mitigates these knowledge gaps, yielding significant performance gains. |
| title | Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.08800 |