Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915295880282112 |
|---|---|
| author | Lin, Yi-Cheng Chen, Wei-Chih Lee, Hung-yi |
| author_facet | Lin, Yi-Cheng Chen, Wei-Chih Lee, Hung-yi |
| contents | Warning: This paper may contain texts with uncomfortable content.
Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07665 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models Lin, Yi-Cheng Chen, Wei-Chih Lee, Hung-yi Computation and Language Audio and Speech Processing Warning: This paper may contain texts with uncomfortable content. Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies. |
| title | Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.07665 |