Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911593299705856 |
|---|---|
| author | Luo, Haozheng Qiu, Chenghao Su, Maojiang Zhou, Zhihan Mehta, Zoe Ye, Guo Hu, Jerry Yao-Chieh Liu, Han |
| author_facet | Luo, Haozheng Qiu, Chenghao Su, Maojiang Zhou, Zhihan Mehta, Zoe Ye, Guo Hu, Jerry Yao-Chieh Liu, Han |
| contents | To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings. Code is available at https://github.com/MAGICS-LAB/GERM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_00598 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fast and Low-Cost Genomic Foundation Models via Outlier Removal Luo, Haozheng Qiu, Chenghao Su, Maojiang Zhou, Zhihan Mehta, Zoe Ye, Guo Hu, Jerry Yao-Chieh Liu, Han Machine Learning Artificial Intelligence To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings. Code is available at https://github.com/MAGICS-LAB/GERM. |
| title | Fast and Low-Cost Genomic Foundation Models via Outlier Removal |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.00598 |