ShiQ: Bringing back Bellman to LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908367170043904 |
|---|---|
| author | Clavier, Pierre Grinsztajn, Nathan Avalos, Raphael Flet-Berliac, Yannis Ergun, Irem Domingues, Omar D. Tarassov, Eugene Pietquin, Olivier Richemond, Pierre H. Strub, Florian Geist, Matthieu |
| author_facet | Clavier, Pierre Grinsztajn, Nathan Avalos, Raphael Flet-Berliac, Yannis Ergun, Irem Domingues, Omar D. Tarassov, Eugene Pietquin, Olivier Richemond, Pierre H. Strub, Florian Geist, Matthieu |
| contents | The fine-tuning of pre-trained large language models (LLMs) using reinforcement learning (RL) is generally formulated as direct policy optimization. This approach was naturally favored as it efficiently improves a pretrained LLM, seen as an initial policy. Another RL paradigm, Q-learning methods, has received far less attention in the LLM community while demonstrating major success in various non-LLM RL tasks. In particular, Q-learning effectiveness comes from its sample efficiency and ability to learn offline, which is particularly valuable given the high computational cost of sampling with LLMs. However, naively applying a Q-learning-style update to the model's logits is ineffective due to the specificity of LLMs. Our core contribution is to derive theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs. To do so, we carefully adapt insights from the RL literature to account for LLM-specific characteristics, ensuring that the logits become reliable Q-value estimates. We then use this loss to build a practical algorithm, ShiQ for Shifted-Q, that supports off-policy, token-wise learning while remaining simple to implement. Finally, we evaluate ShiQ on both synthetic data and real-world benchmarks, e.g., UltraFeedback and BFCL-V3, demonstrating its effectiveness in both single-turn and multi-turn LLM settings |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11081 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ShiQ: Bringing back Bellman to LLMs Clavier, Pierre Grinsztajn, Nathan Avalos, Raphael Flet-Berliac, Yannis Ergun, Irem Domingues, Omar D. Tarassov, Eugene Pietquin, Olivier Richemond, Pierre H. Strub, Florian Geist, Matthieu Machine Learning The fine-tuning of pre-trained large language models (LLMs) using reinforcement learning (RL) is generally formulated as direct policy optimization. This approach was naturally favored as it efficiently improves a pretrained LLM, seen as an initial policy. Another RL paradigm, Q-learning methods, has received far less attention in the LLM community while demonstrating major success in various non-LLM RL tasks. In particular, Q-learning effectiveness comes from its sample efficiency and ability to learn offline, which is particularly valuable given the high computational cost of sampling with LLMs. However, naively applying a Q-learning-style update to the model's logits is ineffective due to the specificity of LLMs. Our core contribution is to derive theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs. To do so, we carefully adapt insights from the RL literature to account for LLM-specific characteristics, ensuring that the logits become reliable Q-value estimates. We then use this loss to build a practical algorithm, ShiQ for Shifted-Q, that supports off-policy, token-wise learning while remaining simple to implement. Finally, we evaluate ShiQ on both synthetic data and real-world benchmarks, e.g., UltraFeedback and BFCL-V3, demonstrating its effectiveness in both single-turn and multi-turn LLM settings |
| title | ShiQ: Bringing back Bellman to LLMs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.11081 |