Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.04282 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915936846479360 |
|---|---|
| author | Wang, Yongyi Li, Lingfeng Chen, Bozhou Li, Ang Liu, Hanyu Zheng, Qirui Yang, Xionghui Li, Wenxin |
| author_facet | Wang, Yongyi Li, Lingfeng Chen, Bozhou Li, Ang Liu, Hanyu Zheng, Qirui Yang, Xionghui Li, Wenxin |
| contents | Recent benchmarks for memory-augmented reinforcement learning (RL) have introduced partially observable Markov decision process (POMDP) environments in which agents must use historical observations to make decisions. However, these benchmarks often lack fine-grained control over the challenges posed to memory models. Synthetic environments offer a solution, enabling precise manipulation of environment dynamics for rigorous and interpretable evaluation of memory-augmented RL. This paper advances the design of such customizable POMDPs with three key contributions: (1) a theoretical framework for analyzing POMDPs based on Memory Demand Structure (MDS) and related concepts; (2) a methodology using linear dynamics, state aggregation, and reward redistribution to construct POMDPs with predefined MDS; and (3) a suite of lightweight, scalable POMDP environments with tunable difficulty, grounded in our theoretical insights. Overall, our work clarifies core challenges in partially observable RL, offers principled guidelines for POMDP design, and aids in selecting and developing suitable memory architectures for RL tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04282 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling Wang, Yongyi Li, Lingfeng Chen, Bozhou Li, Ang Liu, Hanyu Zheng, Qirui Yang, Xionghui Li, Wenxin Artificial Intelligence Recent benchmarks for memory-augmented reinforcement learning (RL) have introduced partially observable Markov decision process (POMDP) environments in which agents must use historical observations to make decisions. However, these benchmarks often lack fine-grained control over the challenges posed to memory models. Synthetic environments offer a solution, enabling precise manipulation of environment dynamics for rigorous and interpretable evaluation of memory-augmented RL. This paper advances the design of such customizable POMDPs with three key contributions: (1) a theoretical framework for analyzing POMDPs based on Memory Demand Structure (MDS) and related concepts; (2) a methodology using linear dynamics, state aggregation, and reward redistribution to construct POMDPs with predefined MDS; and (3) a suite of lightweight, scalable POMDP environments with tunable difficulty, grounded in our theoretical insights. Overall, our work clarifies core challenges in partially observable RL, offers principled guidelines for POMDP design, and aids in selecting and developing suitable memory architectures for RL tasks. |
| title | Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2508.04282 |