Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.21151 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917188640702464 |
|---|---|
| author | Landers, Matthew Killian, Taylor W. Barnes, Hugo Hartvigsen, Thomas Doryab, Afsaneh |
| author_facet | Landers, Matthew Killian, Taylor W. Barnes, Hugo Hartvigsen, Thomas Doryab, Afsaneh |
| contents | Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose Branch Value Estimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to $20\times$ in environments with over four million actions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21151 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces Landers, Matthew Killian, Taylor W. Barnes, Hugo Hartvigsen, Thomas Doryab, Afsaneh Machine Learning Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose Branch Value Estimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to $20\times$ in environments with over four million actions. |
| title | BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.21151 |