BFS-PO: Best-First Search for Large Reasoning Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908836726571008 |
|---|---|
| author | Parascandolo, Fiorenzo Tan, Wenhui Sangineto, Enver Song, Ruihua Cucchiara, Rita |
| author_facet | Parascandolo, Fiorenzo Tan, Wenhui Sangineto, Enver Song, Ruihua Cucchiara, Rita |
| contents | Large Reasoning Models (LRMs) such as OpenAI o1 and DeepSeek-R1 have shown excellent performance in reasoning tasks using long reasoning chains. However, this has also led to a significant increase of computational costs and the generation of verbose output, a phenomenon known as overthinking. The tendency to overthinking is often exacerbated by Reinforcement Learning (RL) algorithms such as GRPO/DAPO. In this paper, we propose BFS-PO, an RL algorithm which alleviates this problem using a Best-First Search exploration strategy. Specifically, BFS-PO looks for the shortest correct answer using a backtracking mechanism based on maximum entropy nodes. By generating progressively shorter responses during training, BFS-PO learns to produce concise reasoning chains. Using different benchmarks and base LRMs, we show that BFS-PO can simultaneously increase the LRM accuracy and shorten its answers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14917 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | BFS-PO: Best-First Search for Large Reasoning Models Parascandolo, Fiorenzo Tan, Wenhui Sangineto, Enver Song, Ruihua Cucchiara, Rita Computation and Language Artificial Intelligence Large Reasoning Models (LRMs) such as OpenAI o1 and DeepSeek-R1 have shown excellent performance in reasoning tasks using long reasoning chains. However, this has also led to a significant increase of computational costs and the generation of verbose output, a phenomenon known as overthinking. The tendency to overthinking is often exacerbated by Reinforcement Learning (RL) algorithms such as GRPO/DAPO. In this paper, we propose BFS-PO, an RL algorithm which alleviates this problem using a Best-First Search exploration strategy. Specifically, BFS-PO looks for the shortest correct answer using a backtracking mechanism based on maximum entropy nodes. By generating progressively shorter responses during training, BFS-PO learns to produce concise reasoning chains. Using different benchmarks and base LRMs, we show that BFS-PO can simultaneously increase the LRM accuracy and shorten its answers. |
| title | BFS-PO: Best-First Search for Large Reasoning Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.14917 |