Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2504.10222 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910910887493632 |
|---|---|
| author | Hu, Pengfei Zhang, Zhenrong Chang, Qikai Liu, Shuhang Ma, Jiefeng Du, Jun Zhang, Jianshu Liu, Quan Gao, Jianqing Ma, Feng Liu, Qingfeng |
| author_facet | Hu, Pengfei Zhang, Zhenrong Chang, Qikai Liu, Shuhang Ma, Jiefeng Du, Jun Zhang, Jianshu Liu, Quan Gao, Jianqing Ma, Feng Liu, Qingfeng |
| contents | Recent work increasingly focuses on improving the reasoning capabilities of Multimodal Large Language Models (MLLMs). Among existing methods, Process Reward Models (PRMs) stand out for offering dense, step-wise supervision to guide intermediate reasoning. However, how to effectively integrate PRMs into search strategies remains an open question. In this paper, we introduce PRM-BAS (PRM-Guided Beam Annealing Search), a lightweight approach for PRM-guided reasoning that dynamically adjusts beam size -- starting with a broader search space and gradually narrowing it as contextual information accumulates, thereby balancing performance and efficiency. We further propose a unified framework for data construction and PRM training. Specifically, we construct the PRM-BAS-300k dataset by selecting 300k questions from existing datasets and performing rollouts at each step to estimate the probability of reaching a correct final answer. The PRM is then trained using a combination of value loss for absolute action quality and rank loss for relative action quality. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate that PRM-BAS significantly improves reasoning performance while maintaining low computational cost. Moreover, it generalizes well across different model scales and architectures, showcasing strong robustness and plug-and-play capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10222 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search Hu, Pengfei Zhang, Zhenrong Chang, Qikai Liu, Shuhang Ma, Jiefeng Du, Jun Zhang, Jianshu Liu, Quan Gao, Jianqing Ma, Feng Liu, Qingfeng Multimedia Recent work increasingly focuses on improving the reasoning capabilities of Multimodal Large Language Models (MLLMs). Among existing methods, Process Reward Models (PRMs) stand out for offering dense, step-wise supervision to guide intermediate reasoning. However, how to effectively integrate PRMs into search strategies remains an open question. In this paper, we introduce PRM-BAS (PRM-Guided Beam Annealing Search), a lightweight approach for PRM-guided reasoning that dynamically adjusts beam size -- starting with a broader search space and gradually narrowing it as contextual information accumulates, thereby balancing performance and efficiency. We further propose a unified framework for data construction and PRM training. Specifically, we construct the PRM-BAS-300k dataset by selecting 300k questions from existing datasets and performing rollouts at each step to estimate the probability of reaching a correct final answer. The PRM is then trained using a combination of value loss for absolute action quality and rank loss for relative action quality. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate that PRM-BAS significantly improves reasoning performance while maintaining low computational cost. Moreover, it generalizes well across different model scales and architectures, showcasing strong robustness and plug-and-play capability. |
| title | PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search |
| topic | Multimedia |
| url | https://arxiv.org/abs/2504.10222 |