AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916617860939776 |
|---|---|
| author | Tan, Xiaoyu Yao, Tianchu Qu, Chao Li, Bin Yang, Minghao Lu, Dakuan Wang, Haozhe Qiu, Xihe Chu, Wei Xu, Yinghui Qi, Yuan |
| author_facet | Tan, Xiaoyu Yao, Tianchu Qu, Chao Li, Bin Yang, Minghao Lu, Dakuan Wang, Haozhe Qiu, Xihe Chu, Wei Xu, Yinghui Qi, Yuan |
| contents | The reasoning capabilities of advanced large language models (LLMs) like o1 have revolutionized artificial intelligence applications. Nevertheless, evaluating and optimizing complex reasoning processes remain significant challenges due to diverse policy distributions and the inherent limitations of human effort and accuracy. In this paper, we present AURORA, a novel automated framework for training universal process reward models (PRMs) using ensemble prompting and reverse verification. The framework employs a two-phase approach: First, it uses diverse prompting strategies and ensemble methods to perform automated annotation and evaluation of processes, ensuring robust assessments for reward learning. Second, it leverages practical reference answers for reverse verification, enhancing the model's ability to validate outputs and improving training accuracy. To assess the framework's performance, we extend beyond the existing ProcessBench benchmark by introducing UniversalBench, which evaluates reward predictions across full trajectories under diverse policy distribtion with long Chain-of-Thought (CoT) outputs. Experimental results demonstrate that AURORA enhances process evaluation accuracy, improves PRMs' accuracy for diverse policy distributions and long-CoT responses. The project will be open-sourced at https://auroraprm.github.io/. The Universal-PRM-7B is available at https://huggingface.co/infly/Universal-PRM-7B. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11520 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification Tan, Xiaoyu Yao, Tianchu Qu, Chao Li, Bin Yang, Minghao Lu, Dakuan Wang, Haozhe Qiu, Xihe Chu, Wei Xu, Yinghui Qi, Yuan Computation and Language The reasoning capabilities of advanced large language models (LLMs) like o1 have revolutionized artificial intelligence applications. Nevertheless, evaluating and optimizing complex reasoning processes remain significant challenges due to diverse policy distributions and the inherent limitations of human effort and accuracy. In this paper, we present AURORA, a novel automated framework for training universal process reward models (PRMs) using ensemble prompting and reverse verification. The framework employs a two-phase approach: First, it uses diverse prompting strategies and ensemble methods to perform automated annotation and evaluation of processes, ensuring robust assessments for reward learning. Second, it leverages practical reference answers for reverse verification, enhancing the model's ability to validate outputs and improving training accuracy. To assess the framework's performance, we extend beyond the existing ProcessBench benchmark by introducing UniversalBench, which evaluates reward predictions across full trajectories under diverse policy distribtion with long Chain-of-Thought (CoT) outputs. Experimental results demonstrate that AURORA enhances process evaluation accuracy, improves PRMs' accuracy for diverse policy distributions and long-CoT responses. The project will be open-sourced at https://auroraprm.github.io/. The Universal-PRM-7B is available at https://huggingface.co/infly/Universal-PRM-7B. |
| title | AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.11520 |