ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909134988771328 |
|---|---|
| author | Zhuo, Le Chi, Zewen Xu, Minghao Huang, Heyan Zheng, Heqi He, Conghui Mao, Xian-Ling Zhang, Wentao |
| author_facet | Zhuo, Le Chi, Zewen Xu, Minghao Huang, Heyan Zheng, Heqi He, Conghui Mao, Xian-Ling Zhang, Wentao |
| contents | We propose ProtLLM, a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the protein-as-word language modeling approach to train ProtLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a large-scale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing ProtLLM with crucial knowledge for understanding proteins. We evaluate ProtLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that ProtLLM not only achieves superior performance against protein-specialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07920 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training Zhuo, Le Chi, Zewen Xu, Minghao Huang, Heyan Zheng, Heqi He, Conghui Mao, Xian-Ling Zhang, Wentao Biomolecules Artificial Intelligence Computation and Language Machine Learning We propose ProtLLM, a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the protein-as-word language modeling approach to train ProtLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a large-scale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing ProtLLM with crucial knowledge for understanding proteins. We evaluate ProtLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that ProtLLM not only achieves superior performance against protein-specialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks. |
| title | ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training |
| topic | Biomolecules Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2403.07920 |