KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918164715012096 |
|---|---|
| author | Ko, Donghyeon Jin, Yeguk Chae, Kyubyung Lee, Byungwook Jo, Chansong In, Sookyo Lee, Jaehong Kim, Taesup Kwak, Donghyun |
| author_facet | Ko, Donghyeon Jin, Yeguk Chae, Kyubyung Lee, Byungwook Jo, Chansong In, Sookyo Lee, Jaehong Kim, Taesup Kwak, Donghyun |
| contents | We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 1,000 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 33.7% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, our analysis of reasoning LLMs shows that engaging reasoning capabilities in the factual QA task can both help models better elicit their latent knowledge and improve their ability to abstain when uncertain. KoSimpleQA can be found at https://anonymous.4open.science/r/KoSimpleQA-62EB. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_18368 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs Ko, Donghyeon Jin, Yeguk Chae, Kyubyung Lee, Byungwook Jo, Chansong In, Sookyo Lee, Jaehong Kim, Taesup Kwak, Donghyun Computation and Language We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 1,000 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 33.7% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, our analysis of reasoning LLMs shows that engaging reasoning capabilities in the factual QA task can both help models better elicit their latent knowledge and improve their ability to abstain when uncertain. KoSimpleQA can be found at https://anonymous.4open.science/r/KoSimpleQA-62EB. |
| title | KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.18368 |