Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910006029320192 |
|---|---|
| author | Xu, Dong Pan, Qihua Yuan, Sisi Li, Jianqiang Zhu, Zexuan Ji, Junkai |
| author_facet | Xu, Dong Pan, Qihua Yuan, Sisi Li, Jianqiang Zhu, Zexuan Ji, Junkai |
| contents | Molecular generative models, often employing GPT-style language modeling on molecular string representations, have shown promising capabilities when scaled to large datasets and model sizes. However, it remains unclear and subject to debate whether these models adhere to predictable scaling laws under fixed computational budgets, which is a crucial understanding for optimally allocating resources between model size, data volume, and molecular representation. In this study, we systematically investigate the scaling behavior of molecular language models across both pretraining and downstream tasks. We train 300 models and conduct over 10,000 experiments, rigorously controlling compute budgets while independently varying model size, number of training tokens, and molecular representation. Our results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer, reveal the substantial impact of molecular representation on performance, and explain previously observed inconsistencies in scaling behavior for molecular generation. Additionally, we publicly release the largest library of molecular language models to date to facilitate future research and development. Code and models are available at https://github.com/SZU-ADDG/MLM-Scaling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22757 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation Xu, Dong Pan, Qihua Yuan, Sisi Li, Jianqiang Zhu, Zexuan Ji, Junkai Machine Learning Biomolecules Molecular generative models, often employing GPT-style language modeling on molecular string representations, have shown promising capabilities when scaled to large datasets and model sizes. However, it remains unclear and subject to debate whether these models adhere to predictable scaling laws under fixed computational budgets, which is a crucial understanding for optimally allocating resources between model size, data volume, and molecular representation. In this study, we systematically investigate the scaling behavior of molecular language models across both pretraining and downstream tasks. We train 300 models and conduct over 10,000 experiments, rigorously controlling compute budgets while independently varying model size, number of training tokens, and molecular representation. Our results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer, reveal the substantial impact of molecular representation on performance, and explain previously observed inconsistencies in scaling behavior for molecular generation. Additionally, we publicly release the largest library of molecular language models to date to facilitate future research and development. Code and models are available at https://github.com/SZU-ADDG/MLM-Scaling. |
| title | Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation |
| topic | Machine Learning Biomolecules |
| url | https://arxiv.org/abs/2601.22757 |