Comparative Study of Multilingual Idioms and Similes in Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917948219719680 |
|---|---|
| author | Khoshtab, Paria Namazifard, Danial Masoudi, Mostafa Akhgary, Ali Sani, Samin Mahdizadeh Yaghoobzadeh, Yadollah |
| author_facet | Khoshtab, Paria Namazifard, Danial Masoudi, Mostafa Akhgary, Ali Sani, Samin Mahdizadeh Yaghoobzadeh, Yadollah |
| contents | This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16461 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparative Study of Multilingual Idioms and Similes in Large Language Models Khoshtab, Paria Namazifard, Danial Masoudi, Mostafa Akhgary, Ali Sani, Samin Mahdizadeh Yaghoobzadeh, Yadollah Computation and Language This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations. |
| title | Comparative Study of Multilingual Idioms and Similes in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.16461 |