Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectives
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916337139318784 |
|---|---|
| author | Wasi, Azmine Toushik Islam, Raima Islam, Mst Rafia Rafi, Taki Hasan Chae, Dong-Kyu |
| author_facet | Wasi, Azmine Toushik Islam, Raima Islam, Mst Rafia Rafi, Taki Hasan Chae, Dong-Kyu |
| contents | While Large Language Models (LLM) have created a massive technological impact in the past decade, allowing for human-enabled applications, they can produce output that contains stereotypes and biases, especially when using low-resource languages. This can be of great ethical concern when dealing with sensitive topics such as religion. As a means toward making LLMS more fair, we explore bias from a religious perspective in Bengali, focusing specifically on two main religious dialects: Hindu and Muslim-majority dialects. Here, we perform different experiments and audit showing the comparative analysis of different sentences using three commonly used LLMs: ChatGPT, Gemini, and Microsoft Copilot, pertaining to the Hindu and Muslim dialects of specific words and showcasing which ones catch the social biases and which do not. Furthermore, we analyze our findings and relate them to potential reasons and evaluation perspectives, considering their global impact with over 300 million speakers worldwide. With this work, we hope to establish the rigor for creating more fairness in LLMs, as these are widely used as creative writing agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_18376 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectives Wasi, Azmine Toushik Islam, Raima Islam, Mst Rafia Rafi, Taki Hasan Chae, Dong-Kyu Human-Computer Interaction Computation and Language Computers and Society Multimedia Social and Information Networks While Large Language Models (LLM) have created a massive technological impact in the past decade, allowing for human-enabled applications, they can produce output that contains stereotypes and biases, especially when using low-resource languages. This can be of great ethical concern when dealing with sensitive topics such as religion. As a means toward making LLMS more fair, we explore bias from a religious perspective in Bengali, focusing specifically on two main religious dialects: Hindu and Muslim-majority dialects. Here, we perform different experiments and audit showing the comparative analysis of different sentences using three commonly used LLMs: ChatGPT, Gemini, and Microsoft Copilot, pertaining to the Hindu and Muslim dialects of specific words and showcasing which ones catch the social biases and which do not. Furthermore, we analyze our findings and relate them to potential reasons and evaluation perspectives, considering their global impact with over 300 million speakers worldwide. With this work, we hope to establish the rigor for creating more fairness in LLMs, as these are widely used as creative writing agents. |
| title | Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectives |
| topic | Human-Computer Interaction Computation and Language Computers and Society Multimedia Social and Information Networks |
| url | https://arxiv.org/abs/2407.18376 |