Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909145826852864 |
|---|---|
| author | Lawrence, Nathan P. Loewen, Philip D. Wang, Shuyuan Forbes, Michael G. Gopaluni, R. Bhushan |
| author_facet | Lawrence, Nathan P. Loewen, Philip D. Wang, Shuyuan Forbes, Michael G. Gopaluni, R. Bhushan |
| contents | We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees provided by using the Youla-Kucera parameterization to define the search domain. Recent advances in behavioral systems allow us to construct a data-driven internal model; this enables an alternative realization of the Youla-Kucera parameterization based entirely on input-output exploration data. Perhaps of independent interest, we formulate and analyze the stability of such data-driven models in the presence of noise. The Youla-Kucera approach requires a stable "parameter" for controller design. For the training of reinforcement learning agents, the set of all stable linear operators is given explicitly through a matrix factorization approach. Moreover, a nonlinear extension is given using a neural network to express a parameterized set of stable operators, which enables seamless integration with standard deep learning libraries. Finally, we show how these ideas can also be applied to tune fixed-structure controllers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_14098 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior Lawrence, Nathan P. Loewen, Philip D. Wang, Shuyuan Forbes, Michael G. Gopaluni, R. Bhushan Machine Learning Artificial Intelligence Systems and Control Optimization and Control We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees provided by using the Youla-Kucera parameterization to define the search domain. Recent advances in behavioral systems allow us to construct a data-driven internal model; this enables an alternative realization of the Youla-Kucera parameterization based entirely on input-output exploration data. Perhaps of independent interest, we formulate and analyze the stability of such data-driven models in the presence of noise. The Youla-Kucera approach requires a stable "parameter" for controller design. For the training of reinforcement learning agents, the set of all stable linear operators is given explicitly through a matrix factorization approach. Moreover, a nonlinear extension is given using a neural network to express a parameterized set of stable operators, which enables seamless integration with standard deep learning libraries. Finally, we show how these ideas can also be applied to tune fixed-structure controllers. |
| title | Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior |
| topic | Machine Learning Artificial Intelligence Systems and Control Optimization and Control |
| url | https://arxiv.org/abs/2310.14098 |