KMMLU: Measuring Massive Multitask Language Understanding in Korean
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929375760351232 |
|---|---|
| author | Son, Guijin Lee, Hanwool Kim, Sungdong Kim, Seungone Muennighoff, Niklas Choi, Taekyoon Park, Cheonbok Yoo, Kang Min Biderman, Stella |
| author_facet | Son, Guijin Lee, Hanwool Kim, Sungdong Kim, Seungone Muennighoff, Niklas Choi, Taekyoon Park, Cheonbok Yoo, Kang Min Biderman, Stella |
| contents | We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean benchmarks are translated from existing English benchmarks, KMMLU is collected from original Korean exams, capturing linguistic and cultural aspects of the Korean language. We test 27 public and proprietary LLMs and observe the best public model to score 50.5%, leaving significant room for improvement. This model was primarily trained for English and Chinese, not Korean. Current LLMs tailored to Korean, such as Polyglot-Ko, perform far worse. Surprisingly, even the most capable proprietary LLMs, e.g., GPT-4 and HyperCLOVA X do not exceed 60%. This suggests that further work is needed to improve LLMs for Korean, and we believe KMMLU offers the appropriate tool to track this progress. We make our dataset publicly available on the Hugging Face Hub and integrate the benchmark into EleutherAI's Language Model Evaluation Harness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11548 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | KMMLU: Measuring Massive Multitask Language Understanding in Korean Son, Guijin Lee, Hanwool Kim, Sungdong Kim, Seungone Muennighoff, Niklas Choi, Taekyoon Park, Cheonbok Yoo, Kang Min Biderman, Stella Computation and Language We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean benchmarks are translated from existing English benchmarks, KMMLU is collected from original Korean exams, capturing linguistic and cultural aspects of the Korean language. We test 27 public and proprietary LLMs and observe the best public model to score 50.5%, leaving significant room for improvement. This model was primarily trained for English and Chinese, not Korean. Current LLMs tailored to Korean, such as Polyglot-Ko, perform far worse. Surprisingly, even the most capable proprietary LLMs, e.g., GPT-4 and HyperCLOVA X do not exceed 60%. This suggests that further work is needed to improve LLMs for Korean, and we believe KMMLU offers the appropriate tool to track this progress. We make our dataset publicly available on the Hugging Face Hub and integrate the benchmark into EleutherAI's Language Model Evaluation Harness. |
| title | KMMLU: Measuring Massive Multitask Language Understanding in Korean |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.11548 |