Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.16570 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908975269675008 |
|---|---|
| author | Tang, Zhijiang Qi, Jiaxin Cui, Yan Ou, Jinli Zheng, Yuhua Huang, Jianqiang |
| author_facet | Tang, Zhijiang Qi, Jiaxin Cui, Yan Ou, Jinli Zheng, Yuhua Huang, Jianqiang |
| contents | DNA sequence encoding is fundamental to gene function prediction, protein synthesis, and diverse downstream biological tasks. Despite the substantial progress achieved by large-scale DNA sequence pretraining, existing studies have overwhelmingly emphasized pretraining scale and custom downstream evaluation datasets, while neglecting some essential components of the pretraining paradigm. In this paper, we reveal three critical yet heretofore overlooked problems in DNA pretraining: inappropriate downstream datasets, inherent flaws in the neighbor-masking strategy, and the lack of detailed discussion on vocabulary. Therefore, we undertake comprehensive investigations and propose principled guidelines, including selection criteria for evaluation datasets, guiding task design, and in-depth vocabulary analysis. Extensive experiments validate the significance of our identified problems and support the rationale behind our recommendations. Finally, we introduce a standardized testbed that enables reproducible and rigorous benchmarking of DNA pretraining methods to advance the development of genomic foundation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_16570 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | In Search of Lost DNA Sequence Pretraining Tang, Zhijiang Qi, Jiaxin Cui, Yan Ou, Jinli Zheng, Yuhua Huang, Jianqiang Machine Learning Artificial Intelligence DNA sequence encoding is fundamental to gene function prediction, protein synthesis, and diverse downstream biological tasks. Despite the substantial progress achieved by large-scale DNA sequence pretraining, existing studies have overwhelmingly emphasized pretraining scale and custom downstream evaluation datasets, while neglecting some essential components of the pretraining paradigm. In this paper, we reveal three critical yet heretofore overlooked problems in DNA pretraining: inappropriate downstream datasets, inherent flaws in the neighbor-masking strategy, and the lack of detailed discussion on vocabulary. Therefore, we undertake comprehensive investigations and propose principled guidelines, including selection criteria for evaluation datasets, guiding task design, and in-depth vocabulary analysis. Extensive experiments validate the significance of our identified problems and support the rationale behind our recommendations. Finally, we introduce a standardized testbed that enables reproducible and rigorous benchmarking of DNA pretraining methods to advance the development of genomic foundation models. |
| title | In Search of Lost DNA Sequence Pretraining |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.16570 |