AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918383894659072 |
|---|---|
| author | Li, Kai Shen, Can Liu, Yile Han, Jirui Zheng, Kelong Zou, Xuechao Wang, Lionel Z. Zhang, Shun Du, Xingjian Luo, Hanjun Jin, Yingbin Xing, Xinxin Ma, Ziyang Liu, Yue Zhang, Yifan Fang, Junfeng Wang, Kun Yan, Yibo Deng, Gelei Li, Haoyang Li, Yiming Zhuang, Xiaobin Chen, Tianlong Wen, Qingsong Zhang, Tianwei Liu, Yang Hu, Haibo Wu, Zhizheng Hu, Xiaolin Chng, Eng-Siong Xu, Wenyuan Wang, XiaoFeng Dong, Wei Li, Xinfeng |
| author_facet | Li, Kai Shen, Can Liu, Yile Han, Jirui Zheng, Kelong Zou, Xuechao Wang, Lionel Z. Zhang, Shun Du, Xingjian Luo, Hanjun Jin, Yingbin Xing, Xinxin Ma, Ziyang Liu, Yue Zhang, Yifan Fang, Junfeng Wang, Kun Yan, Yibo Deng, Gelei Li, Haoyang Li, Yiming Zhuang, Xiaobin Chen, Tianlong Wen, Qingsong Zhang, Tianwei Liu, Yang Hu, Haibo Wu, Zhizheng Hu, Xiaolin Chng, Eng-Siong Xu, Wenyuan Wang, XiaoFeng Dong, Wei Li, Xinfeng |
| contents | The rapid development and widespread adoption of Audio Large Language Models (ALLMs) demand rigorous evaluation of their trustworthiness. However, existing evaluation frameworks are primarily designed for text and fail to capture vulnerabilities introduced by the acoustic properties of audio. We find that significant trustworthiness risks in ALLMs arise from non-semantic acoustic cues, such as timbre, accent, and background noise, which can be exploited to manipulate model behavior. To address this gap, we propose AudioTrust, the first large-scale and systematic framework for evaluating ALLM trustworthiness under audio-specific risks. AudioTrust covers six key dimensions: fairness, hallucination, safety, privacy, robustness, and authenticition. It includes 26 sub-tasks and a curated dataset of more than 4,420 audio samples collected from real-world scenarios, including daily conversations, emergency calls, and voice assistant interactions, and is specifically designed to probe trustworthiness across multiple dimensions. Our comprehensive evaluation spans 18 experimental settings and uses human-validated automated pipelines to enable objective and scalable assessment of model outputs. Experimental results on 14 state-of-the-art open-source and closed-source ALLMs reveal important limitations and failure boundaries under diverse high-risk audio scenarios, providing critical insights for the secure and trustworthy deployment of future audio models. Our platform and benchmark are publicly available at https://github.com/JusperLee/AudioTrust. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16211 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models Li, Kai Shen, Can Liu, Yile Han, Jirui Zheng, Kelong Zou, Xuechao Wang, Lionel Z. Zhang, Shun Du, Xingjian Luo, Hanjun Jin, Yingbin Xing, Xinxin Ma, Ziyang Liu, Yue Zhang, Yifan Fang, Junfeng Wang, Kun Yan, Yibo Deng, Gelei Li, Haoyang Li, Yiming Zhuang, Xiaobin Chen, Tianlong Wen, Qingsong Zhang, Tianwei Liu, Yang Hu, Haibo Wu, Zhizheng Hu, Xiaolin Chng, Eng-Siong Xu, Wenyuan Wang, XiaoFeng Dong, Wei Li, Xinfeng Sound Artificial Intelligence Computation and Language Audio and Speech Processing The rapid development and widespread adoption of Audio Large Language Models (ALLMs) demand rigorous evaluation of their trustworthiness. However, existing evaluation frameworks are primarily designed for text and fail to capture vulnerabilities introduced by the acoustic properties of audio. We find that significant trustworthiness risks in ALLMs arise from non-semantic acoustic cues, such as timbre, accent, and background noise, which can be exploited to manipulate model behavior. To address this gap, we propose AudioTrust, the first large-scale and systematic framework for evaluating ALLM trustworthiness under audio-specific risks. AudioTrust covers six key dimensions: fairness, hallucination, safety, privacy, robustness, and authenticition. It includes 26 sub-tasks and a curated dataset of more than 4,420 audio samples collected from real-world scenarios, including daily conversations, emergency calls, and voice assistant interactions, and is specifically designed to probe trustworthiness across multiple dimensions. Our comprehensive evaluation spans 18 experimental settings and uses human-validated automated pipelines to enable objective and scalable assessment of model outputs. Experimental results on 14 state-of-the-art open-source and closed-source ALLMs reveal important limitations and failure boundaries under diverse high-risk audio scenarios, providing critical insights for the secure and trustworthy deployment of future audio models. Our platform and benchmark are publicly available at https://github.com/JusperLee/AudioTrust. |
| title | AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models |
| topic | Sound Artificial Intelligence Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.16211 |