Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedback
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909201507287040 |
|---|---|
| author | Cassel, Asaf Luo, Haipeng Rosenberg, Aviv Sotnikov, Dmitry |
| author_facet | Cassel, Asaf Luo, Haipeng Rosenberg, Aviv Sotnikov, Dmitry |
| contents | In many real-world applications, it is hard to provide a reward signal in each step of a Reinforcement Learning (RL) process and more natural to give feedback when an episode ends. To this end, we study the recently proposed model of RL with Aggregate Bandit Feedback (RL-ABF), where the agent only observes the sum of rewards at the end of an episode instead of each reward individually. Prior work studied RL-ABF only in tabular settings, where the number of states is assumed to be small. In this paper, we extend ABF to linear function approximation and develop two efficient algorithms with near-optimal regret guarantees: a value-based optimistic algorithm built on a new randomization technique with a Q-functions ensemble, and a policy optimization algorithm that uses a novel hedging scheme over the ensemble. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_07637 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedback Cassel, Asaf Luo, Haipeng Rosenberg, Aviv Sotnikov, Dmitry Machine Learning In many real-world applications, it is hard to provide a reward signal in each step of a Reinforcement Learning (RL) process and more natural to give feedback when an episode ends. To this end, we study the recently proposed model of RL with Aggregate Bandit Feedback (RL-ABF), where the agent only observes the sum of rewards at the end of an episode instead of each reward individually. Prior work studied RL-ABF only in tabular settings, where the number of states is assumed to be small. In this paper, we extend ABF to linear function approximation and develop two efficient algorithms with near-optimal regret guarantees: a value-based optimistic algorithm built on a new randomization technique with a Q-functions ensemble, and a policy optimization algorithm that uses a novel hedging scheme over the ensemble. |
| title | Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedback |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.07637 |