Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.08731 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913862537707520 |
|---|---|
| author | Zollicoffer, Geigh Eaton, Kenneth Balloch, Jonathan Kim, Julia Zhou, Wei Wright, Robert Riedl, Mark O. |
| author_facet | Zollicoffer, Geigh Eaton, Kenneth Balloch, Jonathan Kim, Julia Zhou, Wei Wright, Robert Riedl, Mark O. |
| contents | Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_08731 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Novelty Detection in Reinforcement Learning with World Models Zollicoffer, Geigh Eaton, Kenneth Balloch, Jonathan Kim, Julia Zhou, Wei Wright, Robert Riedl, Mark O. Artificial Intelligence Machine Learning Systems and Control Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms. |
| title | Novelty Detection in Reinforcement Learning with World Models |
| topic | Artificial Intelligence Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2310.08731 |