Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2603.01335 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910037058781184 |
|---|---|
| author | Yu, Tianrun Yang, Yuxiao Wang, Zhaoyang Zhao, Kaixiang Jenkins, Porter Zhang, Xuchao Bansal, Chetan Yao, Huaxiu Zhang, Weitong |
| author_facet | Yu, Tianrun Yang, Yuxiao Wang, Zhaoyang Zhao, Kaixiang Jenkins, Porter Zhang, Xuchao Bansal, Chetan Yao, Huaxiu Zhang, Weitong |
| contents | We study test-time scaling, where a model improves its answer through multi-round self-reflection at inference. We introduce In-Context Policy Optimization (ICPO), in which an agent optimizes its response in context using self-assessed or externally observed rewards without modifying its parameters. To explain this ICPO process, we theoretically show that with sufficient pretraining under a novel Fisher-weighted logit-matching objective, a single-layer linear self-attention model can provably imitate policy-optimization algorithm for linear bandits. Building on this theory, we propose Minimum-Entropy ICPO (ME-ICPO), a practical algorithm that iteratively uses its response and self-assessed reward to refine its response in-context at inference time. By selecting the responses and their rewards with minimum entropy, ME-ICPO ensures the robustness of the self-assessed rewards via majority voting. Across standard mathematical reasoning tasks, ME-ICPO attains competitive, top-tier performance while keeping inference costs affordable compared with other inference-time algorithms. Overall, ICPO provides a principled understanding of self-reflection in LLMs and yields practical benefits for test-time scaling for mathematical reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01335 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Provable and Practical In-Context Policy Optimization for Self-Improvement Yu, Tianrun Yang, Yuxiao Wang, Zhaoyang Zhao, Kaixiang Jenkins, Porter Zhang, Xuchao Bansal, Chetan Yao, Huaxiu Zhang, Weitong Machine Learning Artificial Intelligence We study test-time scaling, where a model improves its answer through multi-round self-reflection at inference. We introduce In-Context Policy Optimization (ICPO), in which an agent optimizes its response in context using self-assessed or externally observed rewards without modifying its parameters. To explain this ICPO process, we theoretically show that with sufficient pretraining under a novel Fisher-weighted logit-matching objective, a single-layer linear self-attention model can provably imitate policy-optimization algorithm for linear bandits. Building on this theory, we propose Minimum-Entropy ICPO (ME-ICPO), a practical algorithm that iteratively uses its response and self-assessed reward to refine its response in-context at inference time. By selecting the responses and their rewards with minimum entropy, ME-ICPO ensures the robustness of the self-assessed rewards via majority voting. Across standard mathematical reasoning tasks, ME-ICPO attains competitive, top-tier performance while keeping inference costs affordable compared with other inference-time algorithms. Overall, ICPO provides a principled understanding of self-reflection in LLMs and yields practical benefits for test-time scaling for mathematical reasoning. |
| title | Provable and Practical In-Context Policy Optimization for Self-Improvement |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.01335 |