HELM: Hierarchical Encoding for mRNA Language Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917952839745536 |
|---|---|
| author | Yazdani-Jahromi, Mehdi Prakash, Mangal Mansi, Tommaso Moskalev, Artem Liao, Rui |
| author_facet | Yazdani-Jahromi, Mehdi Prakash, Mangal Mansi, Tommaso Moskalev, Artem Liao, Rui |
| contents | Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on seven diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12459 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HELM: Hierarchical Encoding for mRNA Language Modeling Yazdani-Jahromi, Mehdi Prakash, Mangal Mansi, Tommaso Moskalev, Artem Liao, Rui Machine Learning Computational Engineering, Finance, and Science Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on seven diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines. |
| title | HELM: Hierarchical Encoding for mRNA Language Modeling |
| topic | Machine Learning Computational Engineering, Finance, and Science |
| url | https://arxiv.org/abs/2410.12459 |