Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912241206427648 |
|---|---|
| author | Kang, Yipeng Wang, Junqi Li, Yexin Wang, Mengmeng Tu, Wenming Wang, Quansen Li, Hengli Wu, Tingjun Feng, Xue Zhong, Fangwei Zheng, Zilong |
| author_facet | Kang, Yipeng Wang, Junqi Li, Yexin Wang, Mengmeng Tu, Wenming Wang, Quansen Li, Hengli Wu, Tingjun Feng, Xue Zhong, Fangwei Zheng, Zilong |
| contents | As large language models (LLMs) become increasingly integrated into critical applications, aligning their behavior with human values presents significant challenges. Current methods, such as Reinforcement Learning from Human Feedback (RLHF), typically focus on a limited set of coarse-grained values and are resource-intensive. Moreover, the correlations between these values remain implicit, leading to unclear explanations for value-steering outcomes. Our work argues that a latent causal value graph underlies the value dimensions of LLMs and that, despite alignment training, this structure remains significantly different from human value systems. We leverage these causal value graphs to guide two lightweight value-steering methods: role-based prompting and sparse autoencoder (SAE) steering, effectively mitigating unexpected side effects. Furthermore, SAE provides a more fine-grained approach to value steering. Experiments on Gemma-2B-IT and Llama3-8B-IT demonstrate the effectiveness and controllability of our methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00581 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective Kang, Yipeng Wang, Junqi Li, Yexin Wang, Mengmeng Tu, Wenming Wang, Quansen Li, Hengli Wu, Tingjun Feng, Xue Zhong, Fangwei Zheng, Zilong Computation and Language Artificial Intelligence Machine Learning As large language models (LLMs) become increasingly integrated into critical applications, aligning their behavior with human values presents significant challenges. Current methods, such as Reinforcement Learning from Human Feedback (RLHF), typically focus on a limited set of coarse-grained values and are resource-intensive. Moreover, the correlations between these values remain implicit, leading to unclear explanations for value-steering outcomes. Our work argues that a latent causal value graph underlies the value dimensions of LLMs and that, despite alignment training, this structure remains significantly different from human value systems. We leverage these causal value graphs to guide two lightweight value-steering methods: role-based prompting and sparse autoencoder (SAE) steering, effectively mitigating unexpected side effects. Furthermore, SAE provides a more fine-grained approach to value steering. Experiments on Gemma-2B-IT and Llama3-8B-IT demonstrate the effectiveness and controllability of our methods. |
| title | Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2501.00581 |