Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.14986 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914766808678400 |
|---|---|
| author | Kläser, Kerstin Banaszewski, Błażej Maddrell-Mander, Samuel McLean, Callum Müller, Luis Parviz, Ali Huang, Shenyang Fitzgibbon, Andrew |
| author_facet | Kläser, Kerstin Banaszewski, Błażej Maddrell-Mander, Samuel McLean, Callum Müller, Luis Parviz, Ali Huang, Shenyang Fitzgibbon, Andrew |
| contents | In biological tasks, data is rarely plentiful as it is generated from hard-to-gather measurements. Therefore, pre-training foundation models on large quantities of available data and then transfer to low-data downstream tasks is a promising direction. However, how to design effective foundation models for molecular learning remains an open question, with existing approaches typically focusing on models with large parameter capacities. In this work, we propose $\texttt{MiniMol}$, a foundational model for molecular learning with 10 million parameters. $\texttt{MiniMol}$ is pre-trained on a mix of roughly 3300 sparsely defined graph- and node-level tasks of both quantum and biological nature. The pre-training dataset includes approximately 6 million molecules and 500 million labels. To demonstrate the generalizability of $\texttt{MiniMol}$ across tasks, we evaluate it on downstream tasks from the Therapeutic Data Commons (TDC) ADMET group showing significant improvements over the prior state-of-the-art foundation model across 17 tasks. $\texttt{MiniMol}$ will be a public and open-sourced model for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14986 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $\texttt{MiniMol}$: A Parameter-Efficient Foundation Model for Molecular Learning Kläser, Kerstin Banaszewski, Błażej Maddrell-Mander, Samuel McLean, Callum Müller, Luis Parviz, Ali Huang, Shenyang Fitzgibbon, Andrew Machine Learning Artificial Intelligence In biological tasks, data is rarely plentiful as it is generated from hard-to-gather measurements. Therefore, pre-training foundation models on large quantities of available data and then transfer to low-data downstream tasks is a promising direction. However, how to design effective foundation models for molecular learning remains an open question, with existing approaches typically focusing on models with large parameter capacities. In this work, we propose $\texttt{MiniMol}$, a foundational model for molecular learning with 10 million parameters. $\texttt{MiniMol}$ is pre-trained on a mix of roughly 3300 sparsely defined graph- and node-level tasks of both quantum and biological nature. The pre-training dataset includes approximately 6 million molecules and 500 million labels. To demonstrate the generalizability of $\texttt{MiniMol}$ across tasks, we evaluate it on downstream tasks from the Therapeutic Data Commons (TDC) ADMET group showing significant improvements over the prior state-of-the-art foundation model across 17 tasks. $\texttt{MiniMol}$ will be a public and open-sourced model for future research. |
| title | $\texttt{MiniMol}$: A Parameter-Efficient Foundation Model for Molecular Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.14986 |