Self-Generated Critiques Boost Reward Modeling for Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917917244784640 |
|---|---|
| author | Yu, Yue Chen, Zhengxing Zhang, Aston Tan, Liang Zhu, Chenguang Pang, Richard Yuanzhe Qian, Yundi Wang, Xuewei Gururangan, Suchin Zhang, Chao Kambadur, Melanie Mahajan, Dhruv Hou, Rui |
| author_facet | Yu, Yue Chen, Zhengxing Zhang, Aston Tan, Liang Zhu, Chenguang Pang, Richard Yuanzhe Qian, Yundi Wang, Xuewei Gururangan, Suchin Zhang, Chao Kambadur, Melanie Mahajan, Dhruv Hou, Rui |
| contents | Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16646 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Generated Critiques Boost Reward Modeling for Language Models Yu, Yue Chen, Zhengxing Zhang, Aston Tan, Liang Zhu, Chenguang Pang, Richard Yuanzhe Qian, Yundi Wang, Xuewei Gururangan, Suchin Zhang, Chao Kambadur, Melanie Mahajan, Dhruv Hou, Rui Computation and Language Artificial Intelligence Machine Learning Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy. |
| title | Self-Generated Critiques Boost Reward Modeling for Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.16646 |