Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915795137724416 |
|---|---|
| author | Lee, Hanwool Choi, Dasol Kim, Sooyong Jeong, Ilgyun Baek, Sangwon Son, Guijin Hwang, Inseon Lee, Naeun Hong, Seunghyeok |
| author_facet | Lee, Hanwool Choi, Dasol Kim, Sooyong Jeong, Ilgyun Baek, Sangwon Son, Guijin Hwang, Inseon Lee, Naeun Hong, Seunghyeok |
| contents | Recent advancements in Korean large language models (LLMs) have driven numerous benchmarks and evaluation methods, yet inconsistent protocols cause up to 10 p.p performance gaps across institutions. Overcoming these reproducibility gaps does not mean enforcing a one-size-fits-all evaluation. Rather, effective benchmarking requires diverse experimental approaches and a framework robust enough to support them. To this end, we introduce HRET (Haerae Evaluation Toolkit), an open-source, registry-based framework that unifies Korean LLM assessment. HRET integrates major Korean benchmarks, multiple inference backends, and multi-method evaluation, with language consistency enforcement to ensure genuine Korean outputs. Its modular registry design also enables rapid incorporation of new datasets, methods, and backends, ensuring the toolkit adapts to evolving research needs. Beyond standard accuracy metrics, HRET incorporates Korean-focused output analyses-morphology-aware Type-Token Ratio (TTR) for evaluating lexical diversity and systematic keyword-omission detection for identifying missing concepts-to provide diagnostic insights into language-specific behaviors. These targeted analyses help researchers pinpoint morphological and semantic shortcomings in model outputs, guiding focused improvements in Korean LLM development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_22968 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Models Lee, Hanwool Choi, Dasol Kim, Sooyong Jeong, Ilgyun Baek, Sangwon Son, Guijin Hwang, Inseon Lee, Naeun Hong, Seunghyeok Computational Engineering, Finance, and Science Artificial Intelligence Computation and Language Recent advancements in Korean large language models (LLMs) have driven numerous benchmarks and evaluation methods, yet inconsistent protocols cause up to 10 p.p performance gaps across institutions. Overcoming these reproducibility gaps does not mean enforcing a one-size-fits-all evaluation. Rather, effective benchmarking requires diverse experimental approaches and a framework robust enough to support them. To this end, we introduce HRET (Haerae Evaluation Toolkit), an open-source, registry-based framework that unifies Korean LLM assessment. HRET integrates major Korean benchmarks, multiple inference backends, and multi-method evaluation, with language consistency enforcement to ensure genuine Korean outputs. Its modular registry design also enables rapid incorporation of new datasets, methods, and backends, ensuring the toolkit adapts to evolving research needs. Beyond standard accuracy metrics, HRET incorporates Korean-focused output analyses-morphology-aware Type-Token Ratio (TTR) for evaluating lexical diversity and systematic keyword-omission detection for identifying missing concepts-to provide diagnostic insights into language-specific behaviors. These targeted analyses help researchers pinpoint morphological and semantic shortcomings in model outputs, guiding focused improvements in Korean LLM development. |
| title | Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Models |
| topic | Computational Engineering, Finance, and Science Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.22968 |