Can Learned Optimization Make Reinforcement Learning Less Difficult?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908319298355200 |
|---|---|
| author | Goldie, Alexander David Lu, Chris Jackson, Matthew Thomas Whiteson, Shimon Foerster, Jakob Nicolaus |
| author_facet | Goldie, Alexander David Lu, Chris Jackson, Matthew Thomas Whiteson, Shimon Foerster, Jakob Nicolaus |
| contents | While reinforcement learning (RL) holds great potential for decision making in the real world, it suffers from a number of unique difficulties which often need specific consideration. In particular: it is highly non-stationary; suffers from high degrees of plasticity loss; and requires exploration to prevent premature convergence to local optima and maximize return. In this paper, we consider whether learned optimization can help overcome these problems. Our method, Learned Optimization for Plasticity, Exploration and Non-stationarity (OPEN), meta-learns an update rule whose input features and output structure are informed by previously proposed solutions to these difficulties. We show that our parameterization is flexible enough to enable meta-learning in diverse learning contexts, including the ability to use stochasticity for exploration. Our experiments demonstrate that when meta-trained on single and small sets of environments, OPEN outperforms or equals traditionally used optimizers. Furthermore, OPEN shows strong generalization characteristics across a range of environments and agent architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07082 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can Learned Optimization Make Reinforcement Learning Less Difficult? Goldie, Alexander David Lu, Chris Jackson, Matthew Thomas Whiteson, Shimon Foerster, Jakob Nicolaus Machine Learning Artificial Intelligence While reinforcement learning (RL) holds great potential for decision making in the real world, it suffers from a number of unique difficulties which often need specific consideration. In particular: it is highly non-stationary; suffers from high degrees of plasticity loss; and requires exploration to prevent premature convergence to local optima and maximize return. In this paper, we consider whether learned optimization can help overcome these problems. Our method, Learned Optimization for Plasticity, Exploration and Non-stationarity (OPEN), meta-learns an update rule whose input features and output structure are informed by previously proposed solutions to these difficulties. We show that our parameterization is flexible enough to enable meta-learning in diverse learning contexts, including the ability to use stochasticity for exploration. Our experiments demonstrate that when meta-trained on single and small sets of environments, OPEN outperforms or equals traditionally used optimizers. Furthermore, OPEN shows strong generalization characteristics across a range of environments and agent architectures. |
| title | Can Learned Optimization Make Reinforcement Learning Less Difficult? |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.07082 |