Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910782594220032 |
|---|---|
| author | de Oliveira, Gabriel Bianchin Pedrini, Helio Dias, Zanoni |
| author_facet | de Oliveira, Gabriel Bianchin Pedrini, Helio Dias, Zanoni |
| contents | Various approaches utilizing Transformer architectures have achieved state-of-the-art results in Natural Language Processing (NLP). Based on this success, numerous architectures have been proposed for other types of data, such as in biology, particularly for protein sequences. Notably among these are the ESM2 architectures, pre-trained on billions of proteins, which form the basis of various state-of-the-art approaches in the field. However, the ESM2 architectures have a limitation regarding input size, restricting it to 1,022 amino acids, which necessitates the use of preprocessing techniques to handle sequences longer than this limit. In this paper, we present the long and quantized versions of the ESM2 architectures, doubling the input size limit to 2,048 amino acids. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_07747 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches de Oliveira, Gabriel Bianchin Pedrini, Helio Dias, Zanoni Machine Learning Quantitative Methods Various approaches utilizing Transformer architectures have achieved state-of-the-art results in Natural Language Processing (NLP). Based on this success, numerous architectures have been proposed for other types of data, such as in biology, particularly for protein sequences. Notably among these are the ESM2 architectures, pre-trained on billions of proteins, which form the basis of various state-of-the-art approaches in the field. However, the ESM2 architectures have a limitation regarding input size, restricting it to 1,022 amino acids, which necessitates the use of preprocessing techniques to handle sequences longer than this limit. In this paper, we present the long and quantized versions of the ESM2 architectures, doubling the input size limit to 2,048 amino acids. |
| title | Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches |
| topic | Machine Learning Quantitative Methods |
| url | https://arxiv.org/abs/2501.07747 |