Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.00628 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918352290578432 |
|---|---|
| author | Lin, Yu-Xiang Li, Chen-An Wei, Sheng-Lun Chen, Po-Chun Chen, Hsin-Hsi Lee, Hung-yi |
| author_facet | Lin, Yu-Xiang Li, Chen-An Wei, Sheng-Lun Chen, Po-Chun Chen, Hsin-Hsi Lee, Hung-yi |
| contents | Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options. An open question is whether their predictions are influenced by the order of answer choices, which would indicate a form of position bias and undermine their reliability. In this paper, we identify and analyze this problem in LALMs. We demonstrate that no model is immune to this bias through extensive experiments on six LALMs across three widely used benchmarks and their spoken counterparts. Shuffling the order of answer options can cause performance fluctuations of up to 24% and even change model rankings, raising concerns about the reliability of current evaluation practices. We also study permutation-based strategies and show that they can mitigate bias in most cases. Our work represents the first systematic investigation of this issue in LALMs, and we hope it raises awareness and motivates further research in this direction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00628 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hearing the Order: Investigating Position Bias in Large Audio-Language Models Lin, Yu-Xiang Li, Chen-An Wei, Sheng-Lun Chen, Po-Chun Chen, Hsin-Hsi Lee, Hung-yi Sound Computation and Language Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options. An open question is whether their predictions are influenced by the order of answer choices, which would indicate a form of position bias and undermine their reliability. In this paper, we identify and analyze this problem in LALMs. We demonstrate that no model is immune to this bias through extensive experiments on six LALMs across three widely used benchmarks and their spoken counterparts. Shuffling the order of answer options can cause performance fluctuations of up to 24% and even change model rankings, raising concerns about the reliability of current evaluation practices. We also study permutation-based strategies and show that they can mitigate bias in most cases. Our work represents the first systematic investigation of this issue in LALMs, and we hope it raises awareness and motivates further research in this direction. |
| title | Hearing the Order: Investigating Position Bias in Large Audio-Language Models |
| topic | Sound Computation and Language |
| url | https://arxiv.org/abs/2510.00628 |