EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908343632658432 |
|---|---|
| author | Qiu, Jiahao He, Yinghui Juan, Xinzhe Wang, Yimin Liu, Yuhan Yao, Zixin Wu, Yue Jiang, Xun Yang, Ling Wang, Mengdi |
| author_facet | Qiu, Jiahao He, Yinghui Juan, Xinzhe Wang, Yimin Liu, Yuhan Yao, Zixin Wu, Yue Jiang, Xun Yang, Ling Wang, Mengdi |
| contents | The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09689 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety Qiu, Jiahao He, Yinghui Juan, Xinzhe Wang, Yimin Liu, Yuhan Yao, Zixin Wu, Yue Jiang, Xun Yang, Ling Wang, Mengdi Artificial Intelligence Computation and Language Computers and Society Human-Computer Interaction Machine Learning The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent |
| title | EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety |
| topic | Artificial Intelligence Computation and Language Computers and Society Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2504.09689 |