From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914071488495616 |
|---|---|
| author | Cao, Hanqun Zhang, Hongrui Xu, Junde Zhang, Zhou Shen, Lingdong Sun, Minghao Liu, Ge Xu, Jinbo Li, Wu-Jun Ni, Jinren de la Fuente-Nunez, Cesar Fu, Tianfan Choi, Yejin Heng, Pheng-Ann Wu, Fang |
| author_facet | Cao, Hanqun Zhang, Hongrui Xu, Junde Zhang, Zhou Shen, Lingdong Sun, Minghao Liu, Ge Xu, Jinbo Li, Wu-Jun Ni, Jinren de la Fuente-Nunez, Cesar Fu, Tianfan Choi, Yejin Heng, Pheng-Ann Wu, Fang |
| contents | Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures. In parallel, reinforcement learning (RL) has broadened exploration and enabled precise multi-objective optimization in protein design. Yet whether RL can push PLMs beyond their pretraining priors to uncover latent sequence-structure-function rules remains unclear. We address this by pairing RL with PLMs across four domains: antimicrobial peptide design, kinase variant optimization, antibody engineering, and inverse folding. Using diverse RL algorithms and model classes, we ask if RL improves sampling efficiency and, more importantly, if it reveals capabilities not captured by supervised learning. Across benchmarks, RL consistently boosts success rates and sample efficiency. Performance follows a three-factor interaction: task headroom, reward fidelity, and policy capacity jointly determine gains. When rewards are accurate and informative, policies have sufficient capacity, and tasks leave room beyond supervised baselines, improvements scale; when rewards are noisy or capacity is constrained, gains saturate despite exploration. This view yields practical guidance for RL in protein design: prioritize reward modeling and calibration before scaling policy size, match algorithm and regularization strength to task difficulty, and allocate capacity where marginal gains are largest. Implementation is available at https://github.com/chq1155/RL-PLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01571 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning? Cao, Hanqun Zhang, Hongrui Xu, Junde Zhang, Zhou Shen, Lingdong Sun, Minghao Liu, Ge Xu, Jinbo Li, Wu-Jun Ni, Jinren de la Fuente-Nunez, Cesar Fu, Tianfan Choi, Yejin Heng, Pheng-Ann Wu, Fang Machine Learning Artificial Intelligence Biomolecules Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures. In parallel, reinforcement learning (RL) has broadened exploration and enabled precise multi-objective optimization in protein design. Yet whether RL can push PLMs beyond their pretraining priors to uncover latent sequence-structure-function rules remains unclear. We address this by pairing RL with PLMs across four domains: antimicrobial peptide design, kinase variant optimization, antibody engineering, and inverse folding. Using diverse RL algorithms and model classes, we ask if RL improves sampling efficiency and, more importantly, if it reveals capabilities not captured by supervised learning. Across benchmarks, RL consistently boosts success rates and sample efficiency. Performance follows a three-factor interaction: task headroom, reward fidelity, and policy capacity jointly determine gains. When rewards are accurate and informative, policies have sufficient capacity, and tasks leave room beyond supervised baselines, improvements scale; when rewards are noisy or capacity is constrained, gains saturate despite exploration. This view yields practical guidance for RL in protein design: prioritize reward modeling and calibration before scaling policy size, match algorithm and regularization strength to task difficulty, and allocate capacity where marginal gains are largest. Implementation is available at https://github.com/chq1155/RL-PLM. |
| title | From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning? |
| topic | Machine Learning Artificial Intelligence Biomolecules |
| url | https://arxiv.org/abs/2510.01571 |