Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916326996443136 |
|---|---|
| author | Wołczyk, Maciej Cupiał, Bartłomiej Ostaszewski, Mateusz Bortkiewicz, Michał Zając, Michał Pascanu, Razvan Kuciński, Łukasz Miłoś, Piotr |
| author_facet | Wołczyk, Maciej Cupiał, Bartłomiej Ostaszewski, Mateusz Bortkiewicz, Michał Zając, Michał Pascanu, Razvan Kuciński, Łukasz Miłoś, Piotr |
| contents | Fine-tuning is a widespread technique that allows practitioners to transfer pre-trained capabilities, as recently showcased by the successful applications of foundation models. However, fine-tuning reinforcement learning (RL) models remains a challenge. This work conceptualizes one specific cause of poor transfer, accentuated in the RL setting by the interplay between actions and observations: forgetting of pre-trained capabilities. Namely, a model deteriorates on the state subspace of the downstream task not visited in the initial phase of fine-tuning, on which the model behaved well due to pre-training. This way, we lose the anticipated transfer benefits. We identify conditions when this problem occurs, showing that it is common and, in many cases, catastrophic. Through a detailed empirical analysis of the challenging NetHack and Montezuma's Revenge environments, we show that standard knowledge retention techniques mitigate the problem and thus allow us to take full advantage of the pre-trained capabilities. In particular, in NetHack, we achieve a new state-of-the-art for neural models, improving the previous best score from $5$K to over $10$K points in the Human Monk scenario. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02868 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem Wołczyk, Maciej Cupiał, Bartłomiej Ostaszewski, Mateusz Bortkiewicz, Michał Zając, Michał Pascanu, Razvan Kuciński, Łukasz Miłoś, Piotr Machine Learning Fine-tuning is a widespread technique that allows practitioners to transfer pre-trained capabilities, as recently showcased by the successful applications of foundation models. However, fine-tuning reinforcement learning (RL) models remains a challenge. This work conceptualizes one specific cause of poor transfer, accentuated in the RL setting by the interplay between actions and observations: forgetting of pre-trained capabilities. Namely, a model deteriorates on the state subspace of the downstream task not visited in the initial phase of fine-tuning, on which the model behaved well due to pre-training. This way, we lose the anticipated transfer benefits. We identify conditions when this problem occurs, showing that it is common and, in many cases, catastrophic. Through a detailed empirical analysis of the challenging NetHack and Montezuma's Revenge environments, we show that standard knowledge retention techniques mitigate the problem and thus allow us to take full advantage of the pre-trained capabilities. In particular, in NetHack, we achieve a new state-of-the-art for neural models, improving the previous best score from $5$K to over $10$K points in the Human Monk scenario. |
| title | Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.02868 |