Meta-Reinforcement Learning with Self-Reflection for Agentic Search
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914404227874816 |
|---|---|
| author | Xiao, Teng Yuan, Yige Ivison, Hamish Zhu, Huaisheng Brahman, Faeze Lambert, Nathan Dasigi, Pradeep Smith, Noah A. Hajishirzi, Hannaneh |
| author_facet | Xiao, Teng Yuan, Yige Ivison, Hamish Zhu, Huaisheng Brahman, Faeze Lambert, Nathan Dasigi, Pradeep Smith, Noah A. Hajishirzi, Hannaneh |
| contents | This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_11327 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Meta-Reinforcement Learning with Self-Reflection for Agentic Search Xiao, Teng Yuan, Yige Ivison, Hamish Zhu, Huaisheng Brahman, Faeze Lambert, Nathan Dasigi, Pradeep Smith, Noah A. Hajishirzi, Hannaneh Machine Learning Computation and Language This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search. |
| title | Meta-Reinforcement Learning with Self-Reflection for Agentic Search |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2603.11327 |