OpenHuEval: Evaluating Large Language Model on Hungarian Specifics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908499739410432 |
|---|---|
| author | Yang, Haote Wei, Xingjian Wu, Jiang Ligeti-Nagy, Noémi Sun, Jiaxing Wang, Yinfan Yang, Zijian Győző Gao, Junyuan Wang, Jingchao Jiang, Bowen Wang, Shasha Yu, Nanjun Zhang, Zihao Hong, Shixin Liu, Hongwei Li, Wei Zhang, Songyang Lin, Dahua Wu, Lijun Prószéky, Gábor He, Conghui |
| author_facet | Yang, Haote Wei, Xingjian Wu, Jiang Ligeti-Nagy, Noémi Sun, Jiaxing Wang, Yinfan Yang, Zijian Győző Gao, Junyuan Wang, Jingchao Jiang, Bowen Wang, Shasha Yu, Nanjun Zhang, Zihao Hong, Shixin Liu, Hongwei Li, Wei Zhang, Songyang Lin, Dahua Wu, Lijun Prószéky, Gábor He, Conghui |
| contents | We introduce OpenHuEval, the first benchmark for LLMs focusing on the Hungarian language and specifics. OpenHuEval is constructed from a vast collection of Hungarian-specific materials sourced from multiple origins. In the construction, we incorporated the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs' generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations. Ultimately, OpenHuEval encompasses eight Hungarian-specific dimensions, featuring five tasks and 3953 questions. Consequently, OpenHuEval provides the comprehensive, in-depth, and scientifically accurate assessment of LLM performance in the context of the Hungarian language and its specifics. We evaluated current mainstream LLMs, including both traditional LLMs and recently developed Large Reasoning Models. The results demonstrate the significant necessity for evaluation and model optimization tailored to the Hungarian language and specifics. We also established the framework for analyzing the thinking processes of LRMs with OpenHuEval, revealing intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example. We will release OpenHuEval at https://github.com/opendatalab/OpenHuEval . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21500 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OpenHuEval: Evaluating Large Language Model on Hungarian Specifics Yang, Haote Wei, Xingjian Wu, Jiang Ligeti-Nagy, Noémi Sun, Jiaxing Wang, Yinfan Yang, Zijian Győző Gao, Junyuan Wang, Jingchao Jiang, Bowen Wang, Shasha Yu, Nanjun Zhang, Zihao Hong, Shixin Liu, Hongwei Li, Wei Zhang, Songyang Lin, Dahua Wu, Lijun Prószéky, Gábor He, Conghui Computation and Language We introduce OpenHuEval, the first benchmark for LLMs focusing on the Hungarian language and specifics. OpenHuEval is constructed from a vast collection of Hungarian-specific materials sourced from multiple origins. In the construction, we incorporated the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs' generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations. Ultimately, OpenHuEval encompasses eight Hungarian-specific dimensions, featuring five tasks and 3953 questions. Consequently, OpenHuEval provides the comprehensive, in-depth, and scientifically accurate assessment of LLM performance in the context of the Hungarian language and its specifics. We evaluated current mainstream LLMs, including both traditional LLMs and recently developed Large Reasoning Models. The results demonstrate the significant necessity for evaluation and model optimization tailored to the Hungarian language and specifics. We also established the framework for analyzing the thinking processes of LRMs with OpenHuEval, revealing intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example. We will release OpenHuEval at https://github.com/opendatalab/OpenHuEval . |
| title | OpenHuEval: Evaluating Large Language Model on Hungarian Specifics |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.21500 |