Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.00438 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917376825491456 |
|---|---|
| author | Jiang, Wenxuan Zuo, Yuxin Zhang, Zijian Wu, Xuecheng Fan, Zining Liu, Wenxuan Chen, Li Li, Xiaoyu Cao, Xuezhi Jin, Xiaolong Liu, Ninghao |
| author_facet | Jiang, Wenxuan Zuo, Yuxin Zhang, Zijian Wu, Xuecheng Fan, Zining Liu, Wenxuan Chen, Li Li, Xiaoyu Cao, Xuezhi Jin, Xiaolong Liu, Ninghao |
| contents | In-Context Reinforcement Learning (ICRL) enables Large Language Models (LLMs) to learn online from external rewards directly within the context window. However, a central challenge in ICRL is reward estimation, as models typically lack access to ground-truths during inference. To address this limitation, we propose Test-Time Rethinking for In-Context Reinforcement Learning (TR-ICRL), a novel ICRL framework designed for both reasoning and knowledge-intensive tasks. TR-ICRL operates by first retrieving the most relevant instances from an unlabeled evaluation set for a given query. During each ICRL iteration, LLM generates a set of candidate answers for every retrieved instance. Next, a pseudo-label is derived from this set through majority voting. This label then serves as a proxy to give reward messages and generate formative feedbacks, guiding LLM through iterative refinement. In the end, this synthesized contextual information is integrated with the original query to form a comprehensive prompt, with the answer determining through a final round of majority voting. TR-ICRL is evaluated on mainstream reasoning and knowledge-intensive tasks, where it demonstrates significant performance gains. Remarkably, TR-ICRL improves Qwen2.5-7B by 21.23% on average on MedQA and even 137.59% on AIME2024. Extensive ablation studies and analyses further validate the effectiveness and robustness of our approach. Our code is available at https://github.com/pangpang-xuan/TR_ICRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_00438 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning Jiang, Wenxuan Zuo, Yuxin Zhang, Zijian Wu, Xuecheng Fan, Zining Liu, Wenxuan Chen, Li Li, Xiaoyu Cao, Xuezhi Jin, Xiaolong Liu, Ninghao Computation and Language In-Context Reinforcement Learning (ICRL) enables Large Language Models (LLMs) to learn online from external rewards directly within the context window. However, a central challenge in ICRL is reward estimation, as models typically lack access to ground-truths during inference. To address this limitation, we propose Test-Time Rethinking for In-Context Reinforcement Learning (TR-ICRL), a novel ICRL framework designed for both reasoning and knowledge-intensive tasks. TR-ICRL operates by first retrieving the most relevant instances from an unlabeled evaluation set for a given query. During each ICRL iteration, LLM generates a set of candidate answers for every retrieved instance. Next, a pseudo-label is derived from this set through majority voting. This label then serves as a proxy to give reward messages and generate formative feedbacks, guiding LLM through iterative refinement. In the end, this synthesized contextual information is integrated with the original query to form a comprehensive prompt, with the answer determining through a final round of majority voting. TR-ICRL is evaluated on mainstream reasoning and knowledge-intensive tasks, where it demonstrates significant performance gains. Remarkably, TR-ICRL improves Qwen2.5-7B by 21.23% on average on MedQA and even 137.59% on AIME2024. Extensive ablation studies and analyses further validate the effectiveness and robustness of our approach. Our code is available at https://github.com/pangpang-xuan/TR_ICRL. |
| title | TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.00438 |