Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914292852326400 |
|---|---|
| author | Kong, Lingkai Satish, Anagha Jiang, Hezi Kangaslahti, Akseli Ma, Andrew Chen, Wenbo Song, Mingxiao Xu, Lily Tambe, Milind |
| author_facet | Kong, Lingkai Satish, Anagha Jiang, Hezi Kangaslahti, Akseli Ma, Andrew Chen, Wenbo Song, Mingxiao Xu, Lily Tambe, Milind |
| contents | Reinforcement learning (RL) with combinatorial action spaces remains challenging because feasible action sets are exponentially large and governed by complex feasibility constraints, making direct policy parameterization impractical. Existing approaches embed task-specific value functions into constrained optimization programs or learn deterministic structured policies, sacrificing generality and policy expressiveness. We propose a solver-induced \emph{latent spherical flow policy} that brings the expressiveness of modern generative policies to combinatorial RL while guaranteeing feasibility by design. Our method, LSFlow, learns a \emph{stochastic} policy in a compact continuous latent space via spherical flow matching, and delegates feasibility to a combinatorial optimization solver that maps each latent sample to a valid structured action. To improve efficiency, we train the value network directly in the latent space, avoiding repeated solver calls during policy optimization. To address the piecewise-constant and discontinuous value landscape induced by solver-based action selection, we introduce a smoothed Bellman operator that yields stable, well-defined learning targets. Empirically, our approach outperforms state-of-the-art baselines by an average of 20.6\% across a range of challenging combinatorial RL tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22211 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actions Kong, Lingkai Satish, Anagha Jiang, Hezi Kangaslahti, Akseli Ma, Andrew Chen, Wenbo Song, Mingxiao Xu, Lily Tambe, Milind Machine Learning Reinforcement learning (RL) with combinatorial action spaces remains challenging because feasible action sets are exponentially large and governed by complex feasibility constraints, making direct policy parameterization impractical. Existing approaches embed task-specific value functions into constrained optimization programs or learn deterministic structured policies, sacrificing generality and policy expressiveness. We propose a solver-induced \emph{latent spherical flow policy} that brings the expressiveness of modern generative policies to combinatorial RL while guaranteeing feasibility by design. Our method, LSFlow, learns a \emph{stochastic} policy in a compact continuous latent space via spherical flow matching, and delegates feasibility to a combinatorial optimization solver that maps each latent sample to a valid structured action. To improve efficiency, we train the value network directly in the latent space, avoiding repeated solver calls during policy optimization. To address the piecewise-constant and discontinuous value landscape induced by solver-based action selection, we introduce a smoothed Bellman operator that yields stable, well-defined learning targets. Empirically, our approach outperforms state-of-the-art baselines by an average of 20.6\% across a range of challenging combinatorial RL tasks. |
| title | Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actions |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.22211 |