Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.06090 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916518720176128 |
|---|---|
| author | Ismail, Aya Abdelsalam Oikarinen, Tuomas Wang, Amy Adebayo, Julius Stanton, Samuel Joren, Taylor Kleinhenz, Joseph Goodman, Allen Bravo, Héctor Corrada Cho, Kyunghyun Frey, Nathan C. |
| author_facet | Ismail, Aya Abdelsalam Oikarinen, Tuomas Wang, Amy Adebayo, Julius Stanton, Samuel Joren, Taylor Kleinhenz, Joseph Goodman, Allen Bravo, Héctor Corrada Cho, Kyunghyun Frey, Nathan C. |
| contents | We introduce Concept Bottleneck Protein Language Models (CB-pLM), a generative masked language model with a layer where each neuron corresponds to an interpretable concept. Our architecture offers three key benefits: i) Control: We can intervene on concept values to precisely control the properties of generated proteins, achieving a 3 times larger change in desired concept values compared to baselines. ii) Interpretability: A linear mapping between concept values and predicted tokens allows transparent analysis of the model's decision-making process. iii) Debugging: This transparency facilitates easy debugging of trained models. Our models achieve pre-training perplexity and downstream task performance comparable to traditional masked protein language models, demonstrating that interpretability does not compromise performance. While adaptable to any language model, we focus on masked protein language models due to their importance in drug discovery and the ability to validate our model's capabilities through real-world experiments and expert knowledge. We scale our CB-pLM from 24 million to 3 billion parameters, making them the largest Concept Bottleneck Models trained and the first capable of generative language modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06090 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Concept Bottleneck Language Models For protein design Ismail, Aya Abdelsalam Oikarinen, Tuomas Wang, Amy Adebayo, Julius Stanton, Samuel Joren, Taylor Kleinhenz, Joseph Goodman, Allen Bravo, Héctor Corrada Cho, Kyunghyun Frey, Nathan C. Machine Learning We introduce Concept Bottleneck Protein Language Models (CB-pLM), a generative masked language model with a layer where each neuron corresponds to an interpretable concept. Our architecture offers three key benefits: i) Control: We can intervene on concept values to precisely control the properties of generated proteins, achieving a 3 times larger change in desired concept values compared to baselines. ii) Interpretability: A linear mapping between concept values and predicted tokens allows transparent analysis of the model's decision-making process. iii) Debugging: This transparency facilitates easy debugging of trained models. Our models achieve pre-training perplexity and downstream task performance comparable to traditional masked protein language models, demonstrating that interpretability does not compromise performance. While adaptable to any language model, we focus on masked protein language models due to their importance in drug discovery and the ability to validate our model's capabilities through real-world experiments and expert knowledge. We scale our CB-pLM from 24 million to 3 billion parameters, making them the largest Concept Bottleneck Models trained and the first capable of generative language modeling. |
| title | Concept Bottleneck Language Models For protein design |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.06090 |