Safe Learning Under Irreversible Dynamics via Asking for Help
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912589334708224 |
|---|---|
| author | Plaut, Benjamin Liévano-Karim, Juan Zhu, Hanlin Russell, Stuart |
| author_facet | Plaut, Benjamin Liévano-Karim, Juan Zhu, Hanlin Russell, Stuart |
| contents | Most learning algorithms with formal regret guarantees essentially rely on trying all possible behaviors, which is problematic when some errors cannot be recovered from. Instead, we allow the learning agent to ask for help from a mentor and to transfer knowledge between similar states. We show that this combination enables the agent to learn both safely and effectively. Under standard online learning assumptions, we provide an algorithm whose regret and number of mentor queries are both sublinear in the time horizon for any Markov Decision Process (MDP), including MDPs with irreversible dynamics. Our proof involves a sequence of three reductions which may be of independent interest. Conceptually, our result may be the first formal proof that it is possible for an agent to obtain high reward while becoming self-sufficient in an unknown, unbounded, and high-stakes environment without resets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14043 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Safe Learning Under Irreversible Dynamics via Asking for Help Plaut, Benjamin Liévano-Karim, Juan Zhu, Hanlin Russell, Stuart Machine Learning Artificial Intelligence Most learning algorithms with formal regret guarantees essentially rely on trying all possible behaviors, which is problematic when some errors cannot be recovered from. Instead, we allow the learning agent to ask for help from a mentor and to transfer knowledge between similar states. We show that this combination enables the agent to learn both safely and effectively. Under standard online learning assumptions, we provide an algorithm whose regret and number of mentor queries are both sublinear in the time horizon for any Markov Decision Process (MDP), including MDPs with irreversible dynamics. Our proof involves a sequence of three reductions which may be of independent interest. Conceptually, our result may be the first formal proof that it is possible for an agent to obtain high reward while becoming self-sufficient in an unknown, unbounded, and high-stakes environment without resets. |
| title | Safe Learning Under Irreversible Dynamics via Asking for Help |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.14043 |