Cover Image

Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Huang, Bingning, Nguyen, Tu, Zimmer, Matthieu
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
by: Li, Shuangtao, et al.
Published: (2025)

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding
by: Liu, Jiacheng, et al.
Published: (2023)

Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning
by: Fei, Wu, et al.
Published: (2025)

Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
by: Ramesh, Shyam Sundhar, et al.
Published: (2026)

Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
by: Maheswaran, Monishwaran, et al.
Published: (2025)

$π^2$: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
by: Do, Quyet V., et al.
Published: (2026)

SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
by: Pham, Thinh, et al.
Published: (2025)

Learning to Reason under Off-Policy Guidance
by: Yan, Jianhao, et al.
Published: (2025)

Mixture of Attentions For Speculative Decoding
by: Zimmer, Matthieu, et al.
Published: (2024)

Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
by: Kunde, Vishnu Teja, et al.
Published: (2026)

SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoning
by: Ai, Zhengyang, et al.
Published: (2026)

Ensembling Language Models with Sequential Monte Carlo
by: Chan, Robin Shing Moon, et al.
Published: (2026)

Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
by: Chen, Peter, et al.
Published: (2025)

Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
by: Liang, Jing, et al.
Published: (2025)

The Model Knows, the Decoder Finds: Future Value Guided Particle Power Sampling
by: Nguyen, Tu, et al.
Published: (2026)

PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning
by: Wu, Feijie, et al.
Published: (2025)

Efficient Reasoning for Large Reasoning Language Models via Certainty-Guided Reflection Suppression
by: Huang, Jiameng, et al.
Published: (2025)

Feedback-Aware Monte Carlo Tree Search for Efficient Information Seeking in Goal-Oriented Conversations
by: Chopra, Harshita, et al.
Published: (2025)

Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
by: Zhao, Stephen, et al.
Published: (2024)

Reasoning Planning for Language Models
by: Nguyen, Bao, et al.
Published: (2025)

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
by: Chegini, Atoosa, et al.
Published: (2026)

No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
by: Le, Thanh-Long V., et al.
Published: (2025)

BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
by: Xi, Zhiheng, et al.
Published: (2025)

Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling
by: Giannone, Giorgio, et al.
Published: (2025)

Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo
by: Loula, João, et al.
Published: (2025)

Advancing LLM Reasoning Generalists with Preference Trees
by: Yuan, Lifan, et al.
Published: (2024)

Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
by: Su, Zhenpeng, et al.
Published: (2025)

Guiding Language Model Reasoning with Planning Tokens
by: Wang, Xinyi, et al.
Published: (2023)

Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A
by: Plaut, Benjamin, et al.
Published: (2024)

GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
by: Li, Sijia, et al.
Published: (2026)

Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives
by: Wang, Zecheng, et al.
Published: (2026)

LLaSA: Large Language and Structured Data Assistant
by: Xu, Yao, et al.
Published: (2024)

HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization
by: Huang, Chengyu, et al.
Published: (2025)

Absolute Zero: Reinforced Self-play Reasoning with Zero Data
by: Zhao, Andrew, et al.
Published: (2025)

Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
by: Gu, Hengrui, et al.
Published: (2026)

Value-Guided Search for Efficient Chain-of-Thought Reasoning
by: Wang, Kaiwen, et al.
Published: (2025)

Dual-Uncertainty Guided Policy Learning for Multimodal Reasoning
by: Liu, Rui, et al.
Published: (2025)

Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
by: Zimmer, Matthieu, et al.
Published: (2025)

PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning
by: Lu, Wenquan, et al.
Published: (2026)

DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
by: Xin, Huajian, et al.
Published: (2024)