Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914131622232064 |
|---|---|
| author | Wang, Fan Shao, Pengtao Zhang, Yiming Yu, Bo Liu, Shaoshan Ding, Ning Cao, Yang Kang, Yu Wang, Haifeng |
| author_facet | Wang, Fan Shao, Pengtao Zhang, Yiming Yu, Bo Liu, Shaoshan Ding, Ning Cao, Yang Kang, Yu Wang, Haifeng |
| contents | In-Context Reinforcement Learning (ICRL) enables agents to learn automatically and on-the-fly from their interactive experiences. However, a major challenge in scaling up ICRL is the lack of scalable task collections. To address this, we propose the procedurally generated tabular Markov Decision Processes, named AnyMDP. Through a carefully designed randomization process, AnyMDP is capable of generating high-quality tasks on a large scale while maintaining relatively low structural biases. To facilitate efficient meta-training at scale, we further introduce decoupled policy distillation and induce prior information in the ICRL framework. Our results demonstrate that, with a sufficiently large scale of AnyMDP tasks, the proposed model can generalize to tasks that were not considered in the training set through versatile in-context learning paradigms. The scalable task set provided by AnyMDP also enables a more thorough empirical investigation of the relationship between data distribution and ICRL performance. We further show that the generalization of ICRL potentially comes at the cost of increased task diversity and longer adaptation periods. This finding carries critical implications for scaling robust ICRL capabilities, highlighting the necessity of diverse and extensive task design, and prioritizing asymptotic performance over few-shot adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02869 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds Wang, Fan Shao, Pengtao Zhang, Yiming Yu, Bo Liu, Shaoshan Ding, Ning Cao, Yang Kang, Yu Wang, Haifeng Machine Learning Artificial Intelligence In-Context Reinforcement Learning (ICRL) enables agents to learn automatically and on-the-fly from their interactive experiences. However, a major challenge in scaling up ICRL is the lack of scalable task collections. To address this, we propose the procedurally generated tabular Markov Decision Processes, named AnyMDP. Through a carefully designed randomization process, AnyMDP is capable of generating high-quality tasks on a large scale while maintaining relatively low structural biases. To facilitate efficient meta-training at scale, we further introduce decoupled policy distillation and induce prior information in the ICRL framework. Our results demonstrate that, with a sufficiently large scale of AnyMDP tasks, the proposed model can generalize to tasks that were not considered in the training set through versatile in-context learning paradigms. The scalable task set provided by AnyMDP also enables a more thorough empirical investigation of the relationship between data distribution and ICRL performance. We further show that the generalization of ICRL potentially comes at the cost of increased task diversity and longer adaptation periods. This finding carries critical implications for scaling robust ICRL capabilities, highlighting the necessity of diverse and extensive task design, and prioritizing asymptotic performance over few-shot adaptation. |
| title | Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.02869 |