Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916494893383680 |
|---|---|
| author | Xi, Zhiheng Yang, Dingwen Huang, Jixuan Tang, Jiafu Li, Guanyu Ding, Yiwen He, Wei Hong, Boyang Do, Shihan Zhan, Wenyu Wang, Xiao Zheng, Rui Ji, Tao Shi, Xiaowei Zhai, Yitao Weng, Rongxiang Wang, Jingang Cai, Xunliang Gui, Tao Wu, Zuxuan Zhang, Qi Qiu, Xipeng Huang, Xuanjing Jiang, Yu-Gang |
| author_facet | Xi, Zhiheng Yang, Dingwen Huang, Jixuan Tang, Jiafu Li, Guanyu Ding, Yiwen He, Wei Hong, Boyang Do, Shihan Zhan, Wenyu Wang, Xiao Zheng, Rui Ji, Tao Shi, Xiaowei Zhai, Yitao Weng, Rongxiang Wang, Jingang Cai, Xunliang Gui, Tao Wu, Zuxuan Zhang, Qi Qiu, Xipeng Huang, Xuanjing Jiang, Yu-Gang |
| contents | Training large language models (LLMs) to spend more time thinking and reflection before responding is crucial for effectively solving complex reasoning tasks in fields such as science, coding, and mathematics. However, the effectiveness of mechanisms like self-reflection and self-correction depends on the model's capacity to accurately assess its own performance, which can be limited by factors such as initial accuracy, question difficulty, and the lack of external feedback. In this paper, we delve into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data, resulting in a dataset of $76,321$ responses paired with step-level feedback. Fine-tuning language models with this dataset enables them to generate natural language feedback for mathematical reasoning. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time, especially when scaling up inference-time computation. Motivated by these findings, we introduce the critique-based supervision to the actor's self-training process, and propose a critique-in-the-loop self-improvement method. Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model. Lastly, we take the preliminary step to explore training self-talk reasoning models via critique supervision and showcase its potential. Our code and datasets are at \href{https://mathcritique.github.io/}{https://mathcritique.github.io/}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16579 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision Xi, Zhiheng Yang, Dingwen Huang, Jixuan Tang, Jiafu Li, Guanyu Ding, Yiwen He, Wei Hong, Boyang Do, Shihan Zhan, Wenyu Wang, Xiao Zheng, Rui Ji, Tao Shi, Xiaowei Zhai, Yitao Weng, Rongxiang Wang, Jingang Cai, Xunliang Gui, Tao Wu, Zuxuan Zhang, Qi Qiu, Xipeng Huang, Xuanjing Jiang, Yu-Gang Computation and Language Artificial Intelligence Machine Learning Training large language models (LLMs) to spend more time thinking and reflection before responding is crucial for effectively solving complex reasoning tasks in fields such as science, coding, and mathematics. However, the effectiveness of mechanisms like self-reflection and self-correction depends on the model's capacity to accurately assess its own performance, which can be limited by factors such as initial accuracy, question difficulty, and the lack of external feedback. In this paper, we delve into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data, resulting in a dataset of $76,321$ responses paired with step-level feedback. Fine-tuning language models with this dataset enables them to generate natural language feedback for mathematical reasoning. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time, especially when scaling up inference-time computation. Motivated by these findings, we introduce the critique-based supervision to the actor's self-training process, and propose a critique-in-the-loop self-improvement method. Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model. Lastly, we take the preliminary step to explore training self-talk reasoning models via critique supervision and showcase its potential. Our code and datasets are at \href{https://mathcritique.github.io/}{https://mathcritique.github.io/}. |
| title | Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.16579 |