Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.14805 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908457120038912 |
|---|---|
| author | Cloud, Alex Le, Minh Chua, James Betley, Jan Sztyber-Betley, Anna Hilton, Jacob Marks, Samuel Evans, Owain |
| author_facet | Cloud, Alex Le, Minh Chua, James Betley, Jan Sztyber-Betley, Anna Hilton, Jacob Marks, Samuel Evans, Owain |
| contents | We study subliminal learning, a surprising phenomenon where language models transmit behavioral traits via semantically unrelated data. In our main experiments, a "teacher" model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a "student" model trained on this dataset learns T. This occurs even when the data is filtered to remove references to T. We observe the same effect when training on code or reasoning traces generated by the same teacher model. However, we do not observe the effect when the teacher and student have different base models. To help explain our findings, we prove a theoretical result showing that subliminal learning occurs in all neural networks under certain conditions, and demonstrate subliminal learning in a simple MLP classifier. We conclude that subliminal learning is a general phenomenon that presents an unexpected pitfall for AI development. Distillation could propagate unintended traits, even when developers try to prevent this via data filtering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14805 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Subliminal Learning: Language models transmit behavioral traits via hidden signals in data Cloud, Alex Le, Minh Chua, James Betley, Jan Sztyber-Betley, Anna Hilton, Jacob Marks, Samuel Evans, Owain Machine Learning Artificial Intelligence We study subliminal learning, a surprising phenomenon where language models transmit behavioral traits via semantically unrelated data. In our main experiments, a "teacher" model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a "student" model trained on this dataset learns T. This occurs even when the data is filtered to remove references to T. We observe the same effect when training on code or reasoning traces generated by the same teacher model. However, we do not observe the effect when the teacher and student have different base models. To help explain our findings, we prove a theoretical result showing that subliminal learning occurs in all neural networks under certain conditions, and demonstrate subliminal learning in a simple MLP classifier. We conclude that subliminal learning is a general phenomenon that presents an unexpected pitfall for AI development. Distillation could propagate unintended traits, even when developers try to prevent this via data filtering. |
| title | Subliminal Learning: Language models transmit behavioral traits via hidden signals in data |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.14805 |