Representation Engineering: A Top-Down Approach to AI Transparency
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916636910419968 |
|---|---|
| author | Zou, Andy Phan, Long Chen, Sarah Campbell, James Guo, Phillip Ren, Richard Pan, Alexander Yin, Xuwang Mazeika, Mantas Dombrowski, Ann-Kathrin Goel, Shashwat Li, Nathaniel Byun, Michael J. Wang, Zifan Mallen, Alex Basart, Steven Koyejo, Sanmi Song, Dawn Fredrikson, Matt Kolter, J. Zico Hendrycks, Dan |
| author_facet | Zou, Andy Phan, Long Chen, Sarah Campbell, James Guo, Phillip Ren, Richard Pan, Alexander Yin, Xuwang Mazeika, Mantas Dombrowski, Ann-Kathrin Goel, Shashwat Li, Nathaniel Byun, Michael J. Wang, Zifan Mallen, Alex Basart, Steven Koyejo, Sanmi Song, Dawn Fredrikson, Matt Kolter, J. Zico Hendrycks, Dan |
| contents | In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_01405 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Representation Engineering: A Top-Down Approach to AI Transparency Zou, Andy Phan, Long Chen, Sarah Campbell, James Guo, Phillip Ren, Richard Pan, Alexander Yin, Xuwang Mazeika, Mantas Dombrowski, Ann-Kathrin Goel, Shashwat Li, Nathaniel Byun, Michael J. Wang, Zifan Mallen, Alex Basart, Steven Koyejo, Sanmi Song, Dawn Fredrikson, Matt Kolter, J. Zico Hendrycks, Dan Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Computers and Society In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems. |
| title | Representation Engineering: A Top-Down Approach to AI Transparency |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Computers and Society |
| url | https://arxiv.org/abs/2310.01405 |