Policy of Thoughts: Scaling LLM Reasoning via Test-time Policy Evolution
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911404281298944 |
|---|---|
| author | Jiao, Zhengbo Xian, Hongyu Wang, Qinglong Ma, Yunpu Wang, Zhebo Zhang, Zifan Kong, Dezhang Han, Meng |
| author_facet | Jiao, Zhengbo Xian, Hongyu Wang, Qinglong Ma, Yunpu Wang, Zhebo Zhang, Zifan Kong, Dezhang Han, Meng |
| contents | Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20379 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Policy of Thoughts: Scaling LLM Reasoning via Test-time Policy Evolution Jiao, Zhengbo Xian, Hongyu Wang, Qinglong Ma, Yunpu Wang, Zhebo Zhang, Zifan Kong, Dezhang Han, Meng Artificial Intelligence Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller. |
| title | Policy of Thoughts: Scaling LLM Reasoning via Test-time Policy Evolution |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.20379 |