Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911876547346432 |
|---|---|
| author | Khan, Muhammad Junaid Ahmed, Syed Hammad Sukthankar, Gita |
| author_facet | Khan, Muhammad Junaid Ahmed, Syed Hammad Sukthankar, Gita |
| contents | We present a novel method aimed at enhancing the sample efficiency of ensemble Q learning. Our proposed approach integrates multi-head self-attention into the ensembled Q networks while bootstrapping the state-action pairs ingested by the ensemble. This not only results in performance improvements over the original REDQ (Chen et al. 2021) and its variant DroQ (Hi-raoka et al. 2022), thereby enhancing Q predictions, but also effectively reduces both the average normalized bias and standard deviation of normalized bias within Q-function ensembles. Importantly, our method also performs well even in scenarios with a low update-to-data (UTD) ratio. Notably, the implementation of our proposed method is straightforward, requiring minimal modifications to the base model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08252 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning Khan, Muhammad Junaid Ahmed, Syed Hammad Sukthankar, Gita Machine Learning Artificial Intelligence We present a novel method aimed at enhancing the sample efficiency of ensemble Q learning. Our proposed approach integrates multi-head self-attention into the ensembled Q networks while bootstrapping the state-action pairs ingested by the ensemble. This not only results in performance improvements over the original REDQ (Chen et al. 2021) and its variant DroQ (Hi-raoka et al. 2022), thereby enhancing Q predictions, but also effectively reduces both the average normalized bias and standard deviation of normalized bias within Q-function ensembles. Importantly, our method also performs well even in scenarios with a low update-to-data (UTD) ratio. Notably, the implementation of our proposed method is straightforward, requiring minimal modifications to the base model. |
| title | Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.08252 |