Twice Sequential Monte Carlo for Tree Search

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Oren, Yaniv, de Vries, Joery A., van der Vaart, Pascal R., Spaan, Matthijs T. J., Böhmer, Wendelin
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866911703847927808
author	Oren, Yaniv de Vries, Joery A. van der Vaart, Pascal R. Spaan, Matthijs T. J. Böhmer, Wendelin
author_facet	Oren, Yaniv de Vries, Joery A. van der Vaart, Pascal R. Spaan, Matthijs T. J. Böhmer, Wendelin
contents	Model-based reinforcement learning (RL) methods that leverage search are responsible for many milestone breakthroughs in RL. Sequential Monte Carlo (SMC) recently emerged as an alternative to the Monte Carlo Tree Search (MCTS) algorithm which drove these breakthroughs. SMC is easier to parallelize and more suitable to GPU acceleration. However, it also suffers from large variance and path degeneracy which prevent it from scaling well with increased search depth, i.e., increased sequential compute. To address these problems, we introduce Twice Sequential Monte Carlo Tree Search (TSMCTS). Across discrete and continuous environments TSMCTS outperforms the SMC baseline as well as a popular modern version of MCTS as a policy improvement operator, scales favorably with sequential compute, reduces estimator variance and mitigates the effects of path degeneracy while retaining the properties that make SMC natural to parallelize.
format	Preprint
id	arxiv_https___arxiv_org_abs_2511_14220
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Twice Sequential Monte Carlo for Tree Search Oren, Yaniv de Vries, Joery A. van der Vaart, Pascal R. Spaan, Matthijs T. J. Böhmer, Wendelin Machine Learning Artificial Intelligence Model-based reinforcement learning (RL) methods that leverage search are responsible for many milestone breakthroughs in RL. Sequential Monte Carlo (SMC) recently emerged as an alternative to the Monte Carlo Tree Search (MCTS) algorithm which drove these breakthroughs. SMC is easier to parallelize and more suitable to GPU acceleration. However, it also suffers from large variance and path degeneracy which prevent it from scaling well with increased search depth, i.e., increased sequential compute. To address these problems, we introduce Twice Sequential Monte Carlo Tree Search (TSMCTS). Across discrete and continuous environments TSMCTS outperforms the SMC baseline as well as a popular modern version of MCTS as a policy improvement operator, scales favorably with sequential compute, reduces estimator variance and mitigates the effects of path degeneracy while retaining the properties that make SMC natural to parallelize.
title	Twice Sequential Monte Carlo for Tree Search
topic	Machine Learning Artificial Intelligence
url	https://arxiv.org/abs/2511.14220