Improving Policy Optimization via $\varepsilon$-Retrain
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915238897516544 |
|---|---|
| author | Marzari, Luca Donti, Priya L. Liu, Changliu Marchesini, Enrico |
| author_facet | Marzari, Luca Donti, Priya L. Liu, Changliu Marchesini, Enrico |
| contents | We present $\varepsilon$-retrain, an exploration strategy encouraging a behavioral preference while optimizing policies with monotonic improvement guarantees. To this end, we introduce an iterative procedure for collecting retrain areas -- parts of the state space where an agent did not satisfy the behavioral preference. Our method switches between the typical uniform restart state distribution and the retrain areas using a decaying factor $\varepsilon$, allowing agents to retrain on situations where they violated the preference. We also employ formal verification of neural networks to provably quantify the degree to which agents adhere to these behavioral preferences. Experiments over hundreds of seeds across locomotion, power network, and navigation tasks show that our method yields agents that exhibit significant performance and sample efficiency improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08315 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Policy Optimization via $\varepsilon$-Retrain Marzari, Luca Donti, Priya L. Liu, Changliu Marchesini, Enrico Artificial Intelligence Machine Learning We present $\varepsilon$-retrain, an exploration strategy encouraging a behavioral preference while optimizing policies with monotonic improvement guarantees. To this end, we introduce an iterative procedure for collecting retrain areas -- parts of the state space where an agent did not satisfy the behavioral preference. Our method switches between the typical uniform restart state distribution and the retrain areas using a decaying factor $\varepsilon$, allowing agents to retrain on situations where they violated the preference. We also employ formal verification of neural networks to provably quantify the degree to which agents adhere to these behavioral preferences. Experiments over hundreds of seeds across locomotion, power network, and navigation tasks show that our method yields agents that exhibit significant performance and sample efficiency improvements. |
| title | Improving Policy Optimization via $\varepsilon$-Retrain |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.08315 |