Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2512.12932 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912889515802624 |
|---|---|
| author | Wu, Yifan Jiang, Jiyue Ye, Xichen Wang, Yiqi Zhou, Chang Xu, Yitao Chen, Jiayang Hu, He Zhang, Weizhong Jin, Cheng Yuan, Jiao Li, Yu |
| author_facet | Wu, Yifan Jiang, Jiyue Ye, Xichen Wang, Yiqi Zhou, Chang Xu, Yitao Chen, Jiayang Hu, He Zhang, Weizhong Jin, Cheng Yuan, Jiao Li, Yu |
| contents | Biological foundation models (BioFMs), pretrained on large-scale biological sequences, have recently shown strong potential in providing meaningful representations for diverse downstream bioinformatics tasks. However, such models often rely on millions to billions of training sequences and billions of parameters, resulting in prohibitive computational costs and significant barriers to reproducibility and accessibility, particularly for academic labs. To address these challenges, we investigate the feasibility of data pruning for BioFM pretraining and propose a post-hoc influence-guided data pruning framework tailored to biological domains. Our approach introduces a subset-based self-influence formulation that enables efficient estimation of sample importance at low computational cost, and builds upon it two simple yet effective selection strategies, namely Top-k Influence (Top I) and Coverage-Centric Influence (CCI). We empirically validate our method on two representative BioFMs, RNA-FM and ESM-C. For RNA, our framework consistently outperforms random selection baselines under an extreme pruning rate of over 99 percent, demonstrating its effectiveness. Furthermore, we show the generalizability of our framework on protein-related tasks using ESM-C. In particular, our coreset even outperforms random subsets that are ten times larger in both RNA and protein settings, revealing substantial redundancy in biological sequence datasets. These findings underscore the potential of influence-guided data pruning to substantially reduce the computational cost of BioFM pretraining, paving the way for more efficient, accessible, and sustainable biological AI research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12932 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating Data Pruning for Pretraining Biological Foundation Models at Scale Wu, Yifan Jiang, Jiyue Ye, Xichen Wang, Yiqi Zhou, Chang Xu, Yitao Chen, Jiayang Hu, He Zhang, Weizhong Jin, Cheng Yuan, Jiao Li, Yu Machine Learning Artificial Intelligence Computational Engineering, Finance, and Science 68T05 I.2.6; J.3 Biological foundation models (BioFMs), pretrained on large-scale biological sequences, have recently shown strong potential in providing meaningful representations for diverse downstream bioinformatics tasks. However, such models often rely on millions to billions of training sequences and billions of parameters, resulting in prohibitive computational costs and significant barriers to reproducibility and accessibility, particularly for academic labs. To address these challenges, we investigate the feasibility of data pruning for BioFM pretraining and propose a post-hoc influence-guided data pruning framework tailored to biological domains. Our approach introduces a subset-based self-influence formulation that enables efficient estimation of sample importance at low computational cost, and builds upon it two simple yet effective selection strategies, namely Top-k Influence (Top I) and Coverage-Centric Influence (CCI). We empirically validate our method on two representative BioFMs, RNA-FM and ESM-C. For RNA, our framework consistently outperforms random selection baselines under an extreme pruning rate of over 99 percent, demonstrating its effectiveness. Furthermore, we show the generalizability of our framework on protein-related tasks using ESM-C. In particular, our coreset even outperforms random subsets that are ten times larger in both RNA and protein settings, revealing substantial redundancy in biological sequence datasets. These findings underscore the potential of influence-guided data pruning to substantially reduce the computational cost of BioFM pretraining, paving the way for more efficient, accessible, and sustainable biological AI research. |
| title | Investigating Data Pruning for Pretraining Biological Foundation Models at Scale |
| topic | Machine Learning Artificial Intelligence Computational Engineering, Finance, and Science 68T05 I.2.6; J.3 |
| url | https://arxiv.org/abs/2512.12932 |