Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911691189518336 |
|---|---|
| author | Lev-Yehudi, Idan Novitsky, Michael Barenboim, Moran Benchetrit, Ron Indelman, Vadim |
| author_facet | Lev-Yehudi, Idan Novitsky, Michael Barenboim, Moran Benchetrit, Ron Indelman, Vadim |
| contents | Online planning in continuous state, action, and observation spaces remains challenging for autonomous systems. While Monte Carlo Tree Search (MCTS) scales effectively via sampling, most continuous (PO)MDP solvers do not exploit gradient-based action optimization. We propose Action-Gradient MCTS (AGMCTS), a framework that combines global tree search with local gradient-based action refinement, while maintaining consistent value estimates. We provide three key theoretical contributions: (1) an action score gradient theorem for particle belief states; (2) the Multiple Importance Sampling (MIS) Tree that supports frequent action-branch updates by reusing prior samples without introducing estimator drift; and (3) tractable action score gradients for smooth generative models using the Area Formula. Empirical results demonstrate that AGMCTS outperforms state-of-the-art sample-based solvers in multiple challenging continuous MDP and POMDP benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12181 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs Lev-Yehudi, Idan Novitsky, Michael Barenboim, Moran Benchetrit, Ron Indelman, Vadim Artificial Intelligence Robotics Online planning in continuous state, action, and observation spaces remains challenging for autonomous systems. While Monte Carlo Tree Search (MCTS) scales effectively via sampling, most continuous (PO)MDP solvers do not exploit gradient-based action optimization. We propose Action-Gradient MCTS (AGMCTS), a framework that combines global tree search with local gradient-based action refinement, while maintaining consistent value estimates. We provide three key theoretical contributions: (1) an action score gradient theorem for particle belief states; (2) the Multiple Importance Sampling (MIS) Tree that supports frequent action-branch updates by reusing prior samples without introducing estimator drift; and (3) tractable action score gradients for smooth generative models using the Area Formula. Empirical results demonstrate that AGMCTS outperforms state-of-the-art sample-based solvers in multiple challenging continuous MDP and POMDP benchmarks. |
| title | Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs |
| topic | Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2503.12181 |