MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909330097307648 |
|---|---|
| author | Park, Dojun Lee, Jiwoo Park, Seohyun Jeong, Hyeyun Koo, Youngeun Hwang, Soonha Park, Seonwoo Lee, Sungeun |
| author_facet | Park, Dojun Lee, Jiwoo Park, Seohyun Jeong, Hyeyun Koo, Youngeun Hwang, Soonha Park, Seonwoo Lee, Sungeun |
| contents | As the capabilities of Large Language Models (LLMs) expand, it becomes increasingly important to evaluate them beyond basic knowledge assessment, focusing on higher-level language understanding. This study introduces MultiPragEval, the first multilingual pragmatic evaluation of LLMs, designed for English, German, Korean, and Chinese. Comprising 1200 question units categorized according to Grice's Cooperative Principle and its four conversational maxims, MultiPragEval enables an in-depth assessment of LLMs' contextual awareness and their ability to infer implied meanings. Our findings demonstrate that Claude3-Opus significantly outperforms other models in all tested languages, establishing a state-of-the-art in the field. Among open-source models, Solar-10.7B and Qwen1.5-14B emerge as strong competitors. By analyzing pragmatic inference, we provide valuable insights into the capabilities essential for advanced language comprehension in AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07736 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models Park, Dojun Lee, Jiwoo Park, Seohyun Jeong, Hyeyun Koo, Youngeun Hwang, Soonha Park, Seonwoo Lee, Sungeun Computation and Language As the capabilities of Large Language Models (LLMs) expand, it becomes increasingly important to evaluate them beyond basic knowledge assessment, focusing on higher-level language understanding. This study introduces MultiPragEval, the first multilingual pragmatic evaluation of LLMs, designed for English, German, Korean, and Chinese. Comprising 1200 question units categorized according to Grice's Cooperative Principle and its four conversational maxims, MultiPragEval enables an in-depth assessment of LLMs' contextual awareness and their ability to infer implied meanings. Our findings demonstrate that Claude3-Opus significantly outperforms other models in all tested languages, establishing a state-of-the-art in the field. Among open-source models, Solar-10.7B and Qwen1.5-14B emerge as strong competitors. By analyzing pragmatic inference, we provide valuable insights into the capabilities essential for advanced language comprehension in AI systems. |
| title | MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.07736 |