Diagnostic Runtime Monitoring with Martingales
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909275087962112 |
|---|---|
| author | Hindy, Ali Luo, Rachel Banerjee, Somrita Kuck, Jonathan Schmerling, Edward Pavone, Marco |
| author_facet | Hindy, Ali Luo, Rachel Banerjee, Somrita Kuck, Jonathan Schmerling, Edward Pavone, Marco |
| contents | Machine learning systems deployed in safety-critical robotics settings must be robust to distribution shifts. However, system designers must understand the cause of a distribution shift in order to implement the appropriate intervention or mitigation strategy and prevent system failure. In this paper, we present a novel framework for diagnosing distribution shifts in a streaming fashion by deploying multiple stochastic martingales simultaneously. We show that knowledge of the underlying cause of a distribution shift can lead to proper interventions over the lifecycle of a deployed system. Our experimental framework can easily be adapted to different types of distribution shifts, models, and datasets. We find that our method outperforms existing work on diagnosing distribution shifts in terms of speed, accuracy, and flexibility, and validate the efficiency of our model in both simulated and live hardware settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_21748 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diagnostic Runtime Monitoring with Martingales Hindy, Ali Luo, Rachel Banerjee, Somrita Kuck, Jonathan Schmerling, Edward Pavone, Marco Robotics Machine Learning Machine learning systems deployed in safety-critical robotics settings must be robust to distribution shifts. However, system designers must understand the cause of a distribution shift in order to implement the appropriate intervention or mitigation strategy and prevent system failure. In this paper, we present a novel framework for diagnosing distribution shifts in a streaming fashion by deploying multiple stochastic martingales simultaneously. We show that knowledge of the underlying cause of a distribution shift can lead to proper interventions over the lifecycle of a deployed system. Our experimental framework can easily be adapted to different types of distribution shifts, models, and datasets. We find that our method outperforms existing work on diagnosing distribution shifts in terms of speed, accuracy, and flexibility, and validate the efficiency of our model in both simulated and live hardware settings. |
| title | Diagnostic Runtime Monitoring with Martingales |
| topic | Robotics Machine Learning |
| url | https://arxiv.org/abs/2407.21748 |