Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.04655 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909195001921536 |
|---|---|
| author | Shen, Siqi Logeswaran, Lajanugen Lee, Moontae Lee, Honglak Poria, Soujanya Mihalcea, Rada |
| author_facet | Shen, Siqi Logeswaran, Lajanugen Lee, Moontae Lee, Honglak Poria, Soujanya Mihalcea, Rada |
| contents | Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs' general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks. Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally aware language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04655 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense Shen, Siqi Logeswaran, Lajanugen Lee, Moontae Lee, Honglak Poria, Soujanya Mihalcea, Rada Computation and Language Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs' general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks. Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally aware language models. |
| title | Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.04655 |