Toward Understanding BERT-Like Pre-Training for DNA Foundation Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917770995695616 |
|---|---|
| author | Liang, Chaoqi Qiao, Lifeng Ye, Peng Dong, Nanqing Sun, Jianle Bai, Weiqiang Ren, Yuchen Ma, Xinzhu Yan, Hongliang Song, Chunfeng Ouyang, Wanli Zuo, Wangmeng |
| author_facet | Liang, Chaoqi Qiao, Lifeng Ye, Peng Dong, Nanqing Sun, Jianle Bai, Weiqiang Ren, Yuchen Ma, Xinzhu Yan, Hongliang Song, Chunfeng Ouyang, Wanli Zuo, Wangmeng |
| contents | With the success of large-scale pre-training in language tasks, there is an increasing trend of applying it to the domain of life sciences. In particular, pre-training methods based on DNA sequences have received increasing attention because of their potential to capture general information about genes. However, existing pre-training methods for DNA sequences largely rely on direct adoptions of BERT pre-training from NLP, lacking a comprehensive understanding and a specifically tailored approach. To address this research gap, we provide the first empirical study with three insightful observations. Based on the empirical study, we notice that overlapping tokenizer can benefit the fine-tuning of downstream tasks but leads to inadequate pre-training with fast convergence. To unleash the pre-training potential, we introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pre-training by continuously expanding its mask boundary, forcing the model to learn more knowledge. RandomMask is simple but effective, achieving state-of-the-art performance across 6 downstream tasks. RandomMask achieves a staggering 68.16\% in Matthew's correlation coefficient for Epigenetic Mark Prediction, a groundbreaking increase of 19.85\% over the baseline and a remarkable 3.69\% improvement over the previous state-of-the-art result. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_07644 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Toward Understanding BERT-Like Pre-Training for DNA Foundation Models Liang, Chaoqi Qiao, Lifeng Ye, Peng Dong, Nanqing Sun, Jianle Bai, Weiqiang Ren, Yuchen Ma, Xinzhu Yan, Hongliang Song, Chunfeng Ouyang, Wanli Zuo, Wangmeng Artificial Intelligence Computation and Language Machine Learning With the success of large-scale pre-training in language tasks, there is an increasing trend of applying it to the domain of life sciences. In particular, pre-training methods based on DNA sequences have received increasing attention because of their potential to capture general information about genes. However, existing pre-training methods for DNA sequences largely rely on direct adoptions of BERT pre-training from NLP, lacking a comprehensive understanding and a specifically tailored approach. To address this research gap, we provide the first empirical study with three insightful observations. Based on the empirical study, we notice that overlapping tokenizer can benefit the fine-tuning of downstream tasks but leads to inadequate pre-training with fast convergence. To unleash the pre-training potential, we introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pre-training by continuously expanding its mask boundary, forcing the model to learn more knowledge. RandomMask is simple but effective, achieving state-of-the-art performance across 6 downstream tasks. RandomMask achieves a staggering 68.16\% in Matthew's correlation coefficient for Epigenetic Mark Prediction, a groundbreaking increase of 19.85\% over the baseline and a remarkable 3.69\% improvement over the previous state-of-the-art result. |
| title | Toward Understanding BERT-Like Pre-Training for DNA Foundation Models |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2310.07644 |