Investigating Memory in Model-Free RL with POPGym Arcade
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917536919977984 |
|---|---|
| author | Wang, Zekang He, Zhe Zhang, Borong Toledo, Edan Morad, Steven |
| author_facet | Wang, Zekang He, Zhe Zhang, Borong Toledo, Edan Morad, Steven |
| contents | How should we analyze memory in deep RL? We introduce tools for analyzing policies under partial observability and revealing how agents use memory to make decisions. To utilize these tools, we present POPGym Arcade, a collection of Atari-inspired, hardware-accelerated environments sharing a single observation and action space. Each environment provides fully and partially observable variants, enabling counterfactual studies on observability. We find that controlled studies are necessary for fair comparisons and identify a pathology where value functions smear credit over irrelevant history. Using this pathology, we demonstrate how out-of-distribution scenarios can contaminate memory, perturbing the policy far into the future. Our code is available at https://github.com/bolt-research/popgym-arcade. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01450 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating Memory in Model-Free RL with POPGym Arcade Wang, Zekang He, Zhe Zhang, Borong Toledo, Edan Morad, Steven Machine Learning Artificial Intelligence Robotics How should we analyze memory in deep RL? We introduce tools for analyzing policies under partial observability and revealing how agents use memory to make decisions. To utilize these tools, we present POPGym Arcade, a collection of Atari-inspired, hardware-accelerated environments sharing a single observation and action space. Each environment provides fully and partially observable variants, enabling counterfactual studies on observability. We find that controlled studies are necessary for fair comparisons and identify a pathology where value functions smear credit over irrelevant history. Using this pathology, we demonstrate how out-of-distribution scenarios can contaminate memory, perturbing the policy far into the future. Our code is available at https://github.com/bolt-research/popgym-arcade. |
| title | Investigating Memory in Model-Free RL with POPGym Arcade |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2503.01450 |