Bridging Offline and Online Reinforcement Learning for LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909660770992128 |
|---|---|
| author | Lanchantin, Jack Chen, Angelica Lan, Janice Li, Xian Saha, Swarnadeep Wang, Tianlu Xu, Jing Yu, Ping Yuan, Weizhe Weston, Jason E Sukhbaatar, Sainbayar Kulikov, Ilia |
| author_facet | Lanchantin, Jack Chen, Angelica Lan, Janice Li, Xian Saha, Swarnadeep Wang, Tianlu Xu, Jing Yu, Ping Yuan, Weizhe Weston, Jason E Sukhbaatar, Sainbayar Kulikov, Ilia |
| contents | We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21495 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bridging Offline and Online Reinforcement Learning for LLMs Lanchantin, Jack Chen, Angelica Lan, Janice Li, Xian Saha, Swarnadeep Wang, Tianlu Xu, Jing Yu, Ping Yuan, Weizhe Weston, Jason E Sukhbaatar, Sainbayar Kulikov, Ilia Computation and Language We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types. |
| title | Bridging Offline and Online Reinforcement Learning for LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.21495 |