Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914913806450688 |
|---|---|
| author | Cheng, Ruoxi Ma, Haoxuan Cao, Shuirong Li, Jiaqi Pei, Aihua Wang, Zhiqiang Ji, Pengliang Wang, Haoyu Huo, Jiaqi |
| author_facet | Cheng, Ruoxi Ma, Haoxuan Cao, Shuirong Li, Jiaqi Pei, Aihua Wang, Zhiqiang Ji, Pengliang Wang, Haoyu Huo, Jiaqi |
| contents | Bias in LLMs can harm user experience and societal outcomes. However, current bias mitigation methods often require intensive human feedback, lack transferability to other topics or yield overconfident and random outputs. We find that involving LLMs in role-playing scenario boosts their ability to recognize and mitigate biases. Based on this, we propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning. Our approach comprises two modes: (1) self-reflection, where the same LLM participates in multi-role debates, and (2) teacher-student, where a more advanced LLM like GPT-3.5-turbo guides the LLM to perform this task. Experimental results across different LLMs on BBQ and our datasets demonstrate the effectiveness of our approach in bias mitigation. Our source code and datasets are available at \texttt{https://anonymous.4open.science/r/RLDF-E344}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10160 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs Cheng, Ruoxi Ma, Haoxuan Cao, Shuirong Li, Jiaqi Pei, Aihua Wang, Zhiqiang Ji, Pengliang Wang, Haoyu Huo, Jiaqi Artificial Intelligence Bias in LLMs can harm user experience and societal outcomes. However, current bias mitigation methods often require intensive human feedback, lack transferability to other topics or yield overconfident and random outputs. We find that involving LLMs in role-playing scenario boosts their ability to recognize and mitigate biases. Based on this, we propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning. Our approach comprises two modes: (1) self-reflection, where the same LLM participates in multi-role debates, and (2) teacher-student, where a more advanced LLM like GPT-3.5-turbo guides the LLM to perform this task. Experimental results across different LLMs on BBQ and our datasets demonstrate the effectiveness of our approach in bias mitigation. Our source code and datasets are available at \texttt{https://anonymous.4open.science/r/RLDF-E344}. |
| title | Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2404.10160 |