HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929718710763520 |
|---|---|
| author | Ma, Mingqian Liu, Guoqing Cao, Chuan Deng, Pan Dao, Tri Gu, Albert Jin, Peiran Yang, Zhao Xia, Yingce Luo, Renqian Hu, Pipi Wang, Zun Chen, Yuan-Jyue Liu, Haiguang Qin, Tao |
| author_facet | Ma, Mingqian Liu, Guoqing Cao, Chuan Deng, Pan Dao, Tri Gu, Albert Jin, Peiran Yang, Zhao Xia, Yingce Luo, Renqian Hu, Pipi Wang, Zun Chen, Yuan-Jyue Liu, Haiguang Qin, Tao |
| contents | Advances in natural language processing and large language models have sparked growing interest in modeling DNA, often referred to as the "language of life". However, DNA modeling poses unique challenges. First, it requires the ability to process ultra-long DNA sequences while preserving single-nucleotide resolution, as individual nucleotides play a critical role in DNA function. Second, success in this domain requires excelling at both generative and understanding tasks: generative tasks hold potential for therapeutic and industrial applications, while understanding tasks provide crucial insights into biological mechanisms and diseases. To address these challenges, we propose HybriDNA, a decoder-only DNA language model that incorporates a hybrid Transformer-Mamba2 architecture, seamlessly integrating the strengths of attention mechanisms with selective state-space models. This hybrid design enables HybriDNA to efficiently process DNA sequences up to 131kb in length with single-nucleotide resolution. HybriDNA achieves state-of-the-art performance across 33 DNA understanding datasets curated from the BEND, GUE, and LRB benchmarks, and demonstrates exceptional capability in generating synthetic cis-regulatory elements (CREs) with desired properties. Furthermore, we show that HybriDNA adheres to expected scaling laws, with performance improving consistently as the model scales from 300M to 3B and 7B parameters. These findings underscore HybriDNA's versatility and its potential to advance DNA research and applications, paving the way for innovations in understanding and engineering the "language of life". |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10807 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model Ma, Mingqian Liu, Guoqing Cao, Chuan Deng, Pan Dao, Tri Gu, Albert Jin, Peiran Yang, Zhao Xia, Yingce Luo, Renqian Hu, Pipi Wang, Zun Chen, Yuan-Jyue Liu, Haiguang Qin, Tao Machine Learning Artificial Intelligence Genomics Advances in natural language processing and large language models have sparked growing interest in modeling DNA, often referred to as the "language of life". However, DNA modeling poses unique challenges. First, it requires the ability to process ultra-long DNA sequences while preserving single-nucleotide resolution, as individual nucleotides play a critical role in DNA function. Second, success in this domain requires excelling at both generative and understanding tasks: generative tasks hold potential for therapeutic and industrial applications, while understanding tasks provide crucial insights into biological mechanisms and diseases. To address these challenges, we propose HybriDNA, a decoder-only DNA language model that incorporates a hybrid Transformer-Mamba2 architecture, seamlessly integrating the strengths of attention mechanisms with selective state-space models. This hybrid design enables HybriDNA to efficiently process DNA sequences up to 131kb in length with single-nucleotide resolution. HybriDNA achieves state-of-the-art performance across 33 DNA understanding datasets curated from the BEND, GUE, and LRB benchmarks, and demonstrates exceptional capability in generating synthetic cis-regulatory elements (CREs) with desired properties. Furthermore, we show that HybriDNA adheres to expected scaling laws, with performance improving consistently as the model scales from 300M to 3B and 7B parameters. These findings underscore HybriDNA's versatility and its potential to advance DNA research and applications, paving the way for innovations in understanding and engineering the "language of life". |
| title | HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model |
| topic | Machine Learning Artificial Intelligence Genomics |
| url | https://arxiv.org/abs/2502.10807 |