Q-function Decomposition with Intervention Semantics with Factored Action Spaces
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916714527064064 |
|---|---|
| author | Lee, Junkyu Gao, Tian Nelson, Elliot Liu, Miao Bhattacharjya, Debarun Lu, Songtao |
| author_facet | Lee, Junkyu Gao, Tian Nelson, Elliot Liu, Miao Bhattacharjya, Debarun Lu, Songtao |
| contents | Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements in sample efficiency compared to state-of-the-art baselines in online continuous control environments and a real-world offline sepsis treatment environment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_21326 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Q-function Decomposition with Intervention Semantics with Factored Action Spaces Lee, Junkyu Gao, Tian Nelson, Elliot Liu, Miao Bhattacharjya, Debarun Lu, Songtao Machine Learning Artificial Intelligence Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements in sample efficiency compared to state-of-the-art baselines in online continuous control environments and a real-world offline sepsis treatment environment. |
| title | Q-function Decomposition with Intervention Semantics with Factored Action Spaces |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2504.21326 |