Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.15368 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910580699299840 |
|---|---|
| author | Khattar, Vanshaj Jin, Ming |
| author_facet | Khattar, Vanshaj Jin, Ming |
| contents | Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_15368 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning Khattar, Vanshaj Jin, Ming Machine Learning Systems and Control Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods. |
| title | Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning |
| topic | Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2408.15368 |