Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918528735510528 |
|---|---|
| author | Lee, Sungyoung Kim, Dohyeong Balachandar, Eshan Mustafaoglu, Zelal Su Pingali, Keshav |
| author_facet | Lee, Sungyoung Kim, Dohyeong Balachandar, Eshan Mustafaoglu, Zelal Su Pingali, Keshav |
| contents | We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_01663 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning Lee, Sungyoung Kim, Dohyeong Balachandar, Eshan Mustafaoglu, Zelal Su Pingali, Keshav Machine Learning Robotics We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN. |
| title | Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning |
| topic | Machine Learning Robotics |
| url | https://arxiv.org/abs/2605.01663 |