Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908617139027968 |
|---|---|
| author | Xi, Zhiheng Huang, Jixuan Guo, Xin Hong, Boyang Yang, Dingwen Fan, Xiaoran Li, Shuo Chen, Zehui Ye, Junjie Yuan, Siyu Du, Zhengyin Yao, Xuesong Xu, Yufei Chen, Jiecao Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Xi, Zhiheng Huang, Jixuan Guo, Xin Hong, Boyang Yang, Dingwen Fan, Xiaoran Li, Shuo Chen, Zehui Ye, Junjie Yuan, Siyu Du, Zhengyin Yao, Xuesong Xu, Yufei Chen, Jiecao Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Training critiquing language models to assess and provide feedback on model outputs is a promising way to improve LLMs for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision. Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor's outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e., determining whether a response is high-quality or not) remains poor, resulting in marginal performance gains. To overcome this, Critique-RL adopts a two-stage optimization strategy. In stage I, it reinforces the discriminability of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique-RL delivers substantial performance improvements. For example, it achieves a 9.02% gain on in-domain tasks and a 5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24320 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning Xi, Zhiheng Huang, Jixuan Guo, Xin Hong, Boyang Yang, Dingwen Fan, Xiaoran Li, Shuo Chen, Zehui Ye, Junjie Yuan, Siyu Du, Zhengyin Yao, Xuesong Xu, Yufei Chen, Jiecao Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Artificial Intelligence Training critiquing language models to assess and provide feedback on model outputs is a promising way to improve LLMs for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision. Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor's outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e., determining whether a response is high-quality or not) remains poor, resulting in marginal performance gains. To overcome this, Critique-RL adopts a two-stage optimization strategy. In stage I, it reinforces the discriminability of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique-RL delivers substantial performance improvements. For example, it achieves a 9.02% gain on in-domain tasks and a 5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential. |
| title | Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.24320 |