RL-finetuning LLMs from on- and off-policy data with a single algorithm
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916665648742400 |
|---|---|
| author | Tang, Yunhao Cohen, Taco Zhang, David W. Valko, Michal Munos, Rémi |
| author_facet | Tang, Yunhao Cohen, Taco Zhang, David W. Valko, Michal Munos, Rémi |
| contents | We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_19612 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RL-finetuning LLMs from on- and off-policy data with a single algorithm Tang, Yunhao Cohen, Taco Zhang, David W. Valko, Michal Munos, Rémi Machine Learning We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms. |
| title | RL-finetuning LLMs from on- and off-policy data with a single algorithm |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.19612 |