Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918175329746944 |
|---|---|
| author | Park, Chanwoo Park, Suyoung Kang, JiA Park, Jongyeon Kim, Sangho Park, Hyunji M. Bae, Sumin Kang, Mingyu Lee, Jaejin |
| author_facet | Park, Chanwoo Park, Suyoung Kang, JiA Park, Jongyeon Kim, Sangho Park, Hyunji M. Bae, Sumin Kang, Mingyu Lee, Jaejin |
| contents | We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean Park, Chanwoo Park, Suyoung Kang, JiA Park, Jongyeon Kim, Sangho Park, Hyunji M. Bae, Sumin Kang, Mingyu Lee, Jaejin Computation and Language Artificial Intelligence We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies. |
| title | Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.24150 |