Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909333172781056 |
|---|---|
| author | Dawson, Fiifi Mosunmola, Zainab Pocker, Sahil Dandekar, Raj Abhijit Dandekar, Rajat Panat, Sreedath |
| author_facet | Dawson, Fiifi Mosunmola, Zainab Pocker, Sahil Dandekar, Raj Abhijit Dandekar, Rajat Panat, Sreedath |
| contents | Although LLMs have been extremely effective in a large number of complex tasks, their understanding and functionality for regional languages and cultures are not well studied. In this paper, we explore the ability of various LLMs to comprehend the cultural aspects of two regional languages: Malayalam (state of Kerala, India) and Yoruba (West Africa). Using Hofstede's six cultural dimensions: Power Distance (PDI), Individualism (IDV), Motivation towards Achievement and Success (MAS), Uncertainty Avoidance (UAV), Long Term Orientation (LTO), and Indulgence (IVR), we quantify the cultural awareness of LLM-based responses. We demonstrate that although LLMs show a high cultural similarity for English, they fail to capture the cultural nuances across these 6 metrics for Malayalam and Yoruba. We also highlight the need for large-scale regional language LLM training with culturally enriched datasets. This will have huge implications for enhancing the user experience of chat-based LLMs and also improving the validity of large-scale LLM agent-based market research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01811 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English Dawson, Fiifi Mosunmola, Zainab Pocker, Sahil Dandekar, Raj Abhijit Dandekar, Rajat Panat, Sreedath Computers and Society Artificial Intelligence Computation and Language Although LLMs have been extremely effective in a large number of complex tasks, their understanding and functionality for regional languages and cultures are not well studied. In this paper, we explore the ability of various LLMs to comprehend the cultural aspects of two regional languages: Malayalam (state of Kerala, India) and Yoruba (West Africa). Using Hofstede's six cultural dimensions: Power Distance (PDI), Individualism (IDV), Motivation towards Achievement and Success (MAS), Uncertainty Avoidance (UAV), Long Term Orientation (LTO), and Indulgence (IVR), we quantify the cultural awareness of LLM-based responses. We demonstrate that although LLMs show a high cultural similarity for English, they fail to capture the cultural nuances across these 6 metrics for Malayalam and Yoruba. We also highlight the need for large-scale regional language LLM training with culturally enriched datasets. This will have huge implications for enhancing the user experience of chat-based LLMs and also improving the validity of large-scale LLM agent-based market research. |
| title | Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English |
| topic | Computers and Society Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.01811 |