BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915331699638272 |
|---|---|
| author | Zhong, Han Yin, Yutong Zhang, Shenao Xu, Xiaojun Liu, Yuanxin Zuo, Yifei Liu, Zhihan Liu, Boyi Zheng, Sirui Guo, Hongyi Wang, Liwei Hong, Mingyi Wang, Zhaoran |
| author_facet | Zhong, Han Yin, Yutong Zhang, Shenao Xu, Xiaojun Liu, Yuanxin Zuo, Yifei Liu, Zhihan Liu, Boyi Zheng, Sirui Guo, Hongyi Wang, Liwei Hong, Mingyi Wang, Zhaoran |
| contents | Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using a novel reward shaping mechanism. Second, it enhances the base LLM by maximizing the joint probability of rationale generation with respect to the model's parameters. Theoretically, we demonstrate BRiTE's convergence at a rate of $1/T$ with $T$ representing the number of iterations. Empirical evaluations on math and coding benchmarks demonstrate that our approach consistently improves performance across different base models without requiring human-annotated thinking processes. In addition, BRiTE demonstrates superior performance compared to existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through supervised fine-tuning with human-annotated data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_18858 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning Zhong, Han Yin, Yutong Zhang, Shenao Xu, Xiaojun Liu, Yuanxin Zuo, Yifei Liu, Zhihan Liu, Boyi Zheng, Sirui Guo, Hongyi Wang, Liwei Hong, Mingyi Wang, Zhaoran Machine Learning Artificial Intelligence Computation and Language Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using a novel reward shaping mechanism. Second, it enhances the base LLM by maximizing the joint probability of rationale generation with respect to the model's parameters. Theoretically, we demonstrate BRiTE's convergence at a rate of $1/T$ with $T$ representing the number of iterations. Empirical evaluations on math and coding benchmarks demonstrate that our approach consistently improves performance across different base models without requiring human-annotated thinking processes. In addition, BRiTE demonstrates superior performance compared to existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through supervised fine-tuning with human-annotated data. |
| title | BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2501.18858 |