Gene42: Long-Range Genomic Foundation Model With Dense Attention
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915212857180160 |
|---|---|
| author | Vishniakov, Kirill Amor, Boulbaba Ben Tekin, Engin ElNaker, Nancy A. Viswanathan, Karthik Medvedev, Aleksandr Singh, Aahan Nadeem, Maryam Sayeed, Mohammad Amaan Kanithi, Praveenkumar Magalhaes, Tiago Vassilieva, Natalia Mahapatra, Dwarikanath Pimentel, Marco Khan, and Shadab |
| author_facet | Vishniakov, Kirill Amor, Boulbaba Ben Tekin, Engin ElNaker, Nancy A. Viswanathan, Karthik Medvedev, Aleksandr Singh, Aahan Nadeem, Maryam Sayeed, Mohammad Amaan Kanithi, Praveenkumar Magalhaes, Tiago Vassilieva, Natalia Mahapatra, Dwarikanath Pimentel, Marco Khan, and Shadab |
| contents | We introduce Gene42, a novel family of Genomic Foundation Models (GFMs) designed to manage context lengths of up to 192,000 base pairs (bp) at a single-nucleotide resolution. Gene42 models utilize a decoder-only (LLaMA-style) architecture with a dense self-attention mechanism. Initially trained on fixed-length sequences of 4,096 bp, our models underwent continuous pretraining to extend the context length to 192,000 bp. This iterative extension allowed for the comprehensive processing of large-scale genomic data and the capture of intricate patterns and dependencies within the human genome. Gene42 is the first dense attention model capable of handling such extensive long context lengths in genomics, challenging state-space models that often rely on convolutional operators among other mechanisms. Our pretrained models exhibit notably low perplexity values and high reconstruction accuracy, highlighting their strong ability to model genomic data. Extensive experiments on various genomic benchmarks have demonstrated state-of-the-art performance across multiple tasks, including biotype classification, regulatory region identification, chromatin profiling prediction, variant pathogenicity prediction, and species classification. The models are publicly available at huggingface.co/inceptionai. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16565 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Gene42: Long-Range Genomic Foundation Model With Dense Attention Vishniakov, Kirill Amor, Boulbaba Ben Tekin, Engin ElNaker, Nancy A. Viswanathan, Karthik Medvedev, Aleksandr Singh, Aahan Nadeem, Maryam Sayeed, Mohammad Amaan Kanithi, Praveenkumar Magalhaes, Tiago Vassilieva, Natalia Mahapatra, Dwarikanath Pimentel, Marco Khan, and Shadab Machine Learning Artificial Intelligence Computation and Language Genomics We introduce Gene42, a novel family of Genomic Foundation Models (GFMs) designed to manage context lengths of up to 192,000 base pairs (bp) at a single-nucleotide resolution. Gene42 models utilize a decoder-only (LLaMA-style) architecture with a dense self-attention mechanism. Initially trained on fixed-length sequences of 4,096 bp, our models underwent continuous pretraining to extend the context length to 192,000 bp. This iterative extension allowed for the comprehensive processing of large-scale genomic data and the capture of intricate patterns and dependencies within the human genome. Gene42 is the first dense attention model capable of handling such extensive long context lengths in genomics, challenging state-space models that often rely on convolutional operators among other mechanisms. Our pretrained models exhibit notably low perplexity values and high reconstruction accuracy, highlighting their strong ability to model genomic data. Extensive experiments on various genomic benchmarks have demonstrated state-of-the-art performance across multiple tasks, including biotype classification, regulatory region identification, chromatin profiling prediction, variant pathogenicity prediction, and species classification. The models are publicly available at huggingface.co/inceptionai. |
| title | Gene42: Long-Range Genomic Foundation Model With Dense Attention |
| topic | Machine Learning Artificial Intelligence Computation and Language Genomics |
| url | https://arxiv.org/abs/2503.16565 |