KoBALT: Korean Benchmark For Advanced Linguistic Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913851965964288 |
|---|---|
| author | Shin, Hyopil Lee, Sangah Jang, Dongjun Song, Wooseok Kim, Jaeyoon Oh, Chaeyoung Jo, Hyemi Ahn, Youngchae Oh, Sihyun Chang, Hyohyeong Kim, Sunkyoung Lee, Jinsik |
| author_facet | Shin, Hyopil Lee, Sangah Jang, Dongjun Song, Wooseok Kim, Jaeyoon Oh, Chaeyoung Jo, Hyemi Ahn, Youngchae Oh, Sihyun Chang, Hyohyeong Kim, Sunkyoung Lee, Jinsik |
| contents | We introduce KoBALT (Korean Benchmark for Advanced Linguistic Tasks), a comprehensive linguistically-motivated benchmark comprising 700 multiple-choice questions spanning 24 phenomena across five linguistic domains: syntax, semantics, pragmatics, phonetics/phonology, and morphology. KoBALT is designed to advance the evaluation of large language models (LLMs) in Korean, a morphologically rich language, by addressing the limitations of conventional benchmarks that often lack linguistic depth and typological grounding. It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora, substantially mitigating the risk of data contamination and allowing a more robust assessment of true language understanding. Our evaluation of 20 contemporary LLMs reveals significant performance disparities, with the highest-performing model achieving 61\% general accuracy but showing substantial variation across linguistic domains - from stronger performance in semantics (66\%) to considerable weaknesses in phonology (31\%) and morphology (36\%). Through human preference evaluation with 95 annotators, we demonstrate a strong correlation between KoBALT scores and human judgments, validating our benchmark's effectiveness as a discriminative measure of Korean language understanding. KoBALT addresses critical gaps in linguistic evaluation for typologically diverse languages and provides a robust framework for assessing genuine linguistic competence in Korean language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16125 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KoBALT: Korean Benchmark For Advanced Linguistic Tasks Shin, Hyopil Lee, Sangah Jang, Dongjun Song, Wooseok Kim, Jaeyoon Oh, Chaeyoung Jo, Hyemi Ahn, Youngchae Oh, Sihyun Chang, Hyohyeong Kim, Sunkyoung Lee, Jinsik Computation and Language We introduce KoBALT (Korean Benchmark for Advanced Linguistic Tasks), a comprehensive linguistically-motivated benchmark comprising 700 multiple-choice questions spanning 24 phenomena across five linguistic domains: syntax, semantics, pragmatics, phonetics/phonology, and morphology. KoBALT is designed to advance the evaluation of large language models (LLMs) in Korean, a morphologically rich language, by addressing the limitations of conventional benchmarks that often lack linguistic depth and typological grounding. It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora, substantially mitigating the risk of data contamination and allowing a more robust assessment of true language understanding. Our evaluation of 20 contemporary LLMs reveals significant performance disparities, with the highest-performing model achieving 61\% general accuracy but showing substantial variation across linguistic domains - from stronger performance in semantics (66\%) to considerable weaknesses in phonology (31\%) and morphology (36\%). Through human preference evaluation with 95 annotators, we demonstrate a strong correlation between KoBALT scores and human judgments, validating our benchmark's effectiveness as a discriminative measure of Korean language understanding. KoBALT addresses critical gaps in linguistic evaluation for typologically diverse languages and provides a robust framework for assessing genuine linguistic competence in Korean language models. |
| title | KoBALT: Korean Benchmark For Advanced Linguistic Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.16125 |