Learning to Rank Chain-of-Thought: Using a Small Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912619757043712 |
|---|---|
| author | Jiang, Eric Hanchen Luo, Haozheng Pang, Shengyuan Li, Xiaomin Qi, Zhenting Li, Hengli Yang, Cheng-Fu Lin, Zongyu Li, Xinfeng Xu, Hao Chang, Kai-Wei Wu, Ying Nian |
| author_facet | Jiang, Eric Hanchen Luo, Haozheng Pang, Shengyuan Li, Xiaomin Qi, Zhenting Li, Hengli Yang, Cheng-Fu Lin, Zongyu Li, Xinfeng Xu, Hao Chang, Kai-Wei Wu, Ying Nian |
| contents | Large Language Models (LLMs) struggle with reliable mathematical reasoning, and current verification methods are often computationally expensive. This paper introduces the Energy Outcome Reward Model (EORM), a highly efficient, lightweight post-hoc verifier designed to address this challenge. EORM uses an energy-based framework to rank Chain-of-Thought (CoT) solutions, learning to distinguish correct from incorrect reasoning using only simple outcome labels, thus eliminating the need for expensive annotations. With only 55M parameters, over 127 times smaller than typical reward models, EORM boosts the accuracy of Llama 3 8B to 90.7\% on GSM8k and 63.7\% on MATH. This performance is achieved by efficiently selecting the optimal reasoning path from a pool of candidates, allowing it to match or exceed the accuracy of far more resource-intensive Best-of-N sampling techniques. Crucially, our experiments show that EORM generalizes effectively to out-of-distribution problems and unseen models, indicating it learns fundamental principles of valid reasoning. This robustness, combined with its efficiency, establishes EORM as a practical tool for deploying more dependable LLMs in complex, real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14999 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning to Rank Chain-of-Thought: Using a Small Model Jiang, Eric Hanchen Luo, Haozheng Pang, Shengyuan Li, Xiaomin Qi, Zhenting Li, Hengli Yang, Cheng-Fu Lin, Zongyu Li, Xinfeng Xu, Hao Chang, Kai-Wei Wu, Ying Nian Machine Learning Artificial Intelligence Computation and Language Large Language Models (LLMs) struggle with reliable mathematical reasoning, and current verification methods are often computationally expensive. This paper introduces the Energy Outcome Reward Model (EORM), a highly efficient, lightweight post-hoc verifier designed to address this challenge. EORM uses an energy-based framework to rank Chain-of-Thought (CoT) solutions, learning to distinguish correct from incorrect reasoning using only simple outcome labels, thus eliminating the need for expensive annotations. With only 55M parameters, over 127 times smaller than typical reward models, EORM boosts the accuracy of Llama 3 8B to 90.7\% on GSM8k and 63.7\% on MATH. This performance is achieved by efficiently selecting the optimal reasoning path from a pool of candidates, allowing it to match or exceed the accuracy of far more resource-intensive Best-of-N sampling techniques. Crucially, our experiments show that EORM generalizes effectively to out-of-distribution problems and unseen models, indicating it learns fundamental principles of valid reasoning. This robustness, combined with its efficiency, establishes EORM as a practical tool for deploying more dependable LLMs in complex, real-world applications. |
| title | Learning to Rank Chain-of-Thought: Using a Small Model |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.14999 |