Online Distribution Shift Detection via Recency Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917669195743232 |
|---|---|
| author | Luo, Rachel Sinha, Rohan Sun, Yixiao Hindy, Ali Zhao, Shengjia Savarese, Silvio Schmerling, Edward Pavone, Marco |
| author_facet | Luo, Rachel Sinha, Rohan Sun, Yixiao Hindy, Ali Zhao, Shengjia Savarese, Silvio Schmerling, Edward Pavone, Marco |
| contents | When deploying modern machine learning-enabled robotic systems in high-stakes applications, detecting distribution shift is critical. However, most existing methods for detecting distribution shift are not well-suited to robotics settings, where data often arrives in a streaming fashion and may be very high-dimensional. In this work, we present an online method for detecting distribution shift with guarantees on the false positive rate - i.e., when there is no distribution shift, our system is very unlikely (with probability $< ε$) to falsely issue an alert; any alerts that are issued should therefore be heeded. Our method is specifically designed for efficient detection even with high dimensional data, and it empirically achieves up to 11x faster detection on realistic robotics settings compared to prior work while maintaining a low false negative rate in practice (whenever there is a distribution shift in our experiments, our method indeed emits an alert). We demonstrate our approach in both simulation and hardware for a visual servoing task, and show that our method indeed issues an alert before a failure occurs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_09916 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Online Distribution Shift Detection via Recency Prediction Luo, Rachel Sinha, Rohan Sun, Yixiao Hindy, Ali Zhao, Shengjia Savarese, Silvio Schmerling, Edward Pavone, Marco Robotics Machine Learning When deploying modern machine learning-enabled robotic systems in high-stakes applications, detecting distribution shift is critical. However, most existing methods for detecting distribution shift are not well-suited to robotics settings, where data often arrives in a streaming fashion and may be very high-dimensional. In this work, we present an online method for detecting distribution shift with guarantees on the false positive rate - i.e., when there is no distribution shift, our system is very unlikely (with probability $< ε$) to falsely issue an alert; any alerts that are issued should therefore be heeded. Our method is specifically designed for efficient detection even with high dimensional data, and it empirically achieves up to 11x faster detection on realistic robotics settings compared to prior work while maintaining a low false negative rate in practice (whenever there is a distribution shift in our experiments, our method indeed emits an alert). We demonstrate our approach in both simulation and hardware for a visual servoing task, and show that our method indeed issues an alert before a failure occurs. |
| title | Online Distribution Shift Detection via Recency Prediction |
| topic | Robotics Machine Learning |
| url | https://arxiv.org/abs/2211.09916 |