Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929501245538304 |
|---|---|
| author | Trella, Anna L. Ghosh, Susobhan Bonar, Erin E. Coughlin, Lara Doshi-Velez, Finale Guo, Yongyi Hung, Pei-Yao Nahum-Shani, Inbal Shetty, Vivek Walton, Maureen Yan, Iris Zhang, Kelly W. Murphy, Susan A. |
| author_facet | Trella, Anna L. Ghosh, Susobhan Bonar, Erin E. Coughlin, Lara Doshi-Velez, Finale Guo, Yongyi Hung, Pei-Yao Nahum-Shani, Inbal Shetty, Vivek Walton, Maureen Yan, Iris Zhang, Kelly W. Murphy, Susan A. |
| contents | Online AI decision-making algorithms are increasingly used by digital interventions to dynamically personalize treatment to individuals. These algorithms determine, in real-time, the delivery of treatment based on accruing data. The objective of this paper is to provide guidelines for enabling effective monitoring of online decision-making algorithms with the goal of (1) safeguarding individuals and (2) ensuring data quality. We elucidate guidelines and discuss our experience in monitoring online decision-making algorithms in two digital intervention clinical trials (Oralytics and MiWaves). Our guidelines include (1) developing fallback methods, pre-specified procedures executed when an issue occurs, and (2) identifying potential issues categorizing them by severity (red, yellow, and green). Across both trials, the monitoring systems detected real-time issues such as out-of-memory issues, database timeout, and failed communication with an external source. Fallback methods prevented participants from not receiving any treatment during the trial and also prevented the use of incorrect data in statistical analyses. These trials provide case studies for how health scientists can build monitoring systems for their digital intervention. Without these algorithm monitoring systems, critical issues would have gone undetected and unresolved. Instead, these monitoring systems safeguarded participants and ensured the quality of the resulting data for updating the intervention and facilitating scientific discovery. These monitoring guidelines and findings give digital intervention teams the confidence to include online decision-making algorithms in digital interventions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10526 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation Trella, Anna L. Ghosh, Susobhan Bonar, Erin E. Coughlin, Lara Doshi-Velez, Finale Guo, Yongyi Hung, Pei-Yao Nahum-Shani, Inbal Shetty, Vivek Walton, Maureen Yan, Iris Zhang, Kelly W. Murphy, Susan A. Computers and Society Artificial Intelligence Online AI decision-making algorithms are increasingly used by digital interventions to dynamically personalize treatment to individuals. These algorithms determine, in real-time, the delivery of treatment based on accruing data. The objective of this paper is to provide guidelines for enabling effective monitoring of online decision-making algorithms with the goal of (1) safeguarding individuals and (2) ensuring data quality. We elucidate guidelines and discuss our experience in monitoring online decision-making algorithms in two digital intervention clinical trials (Oralytics and MiWaves). Our guidelines include (1) developing fallback methods, pre-specified procedures executed when an issue occurs, and (2) identifying potential issues categorizing them by severity (red, yellow, and green). Across both trials, the monitoring systems detected real-time issues such as out-of-memory issues, database timeout, and failed communication with an external source. Fallback methods prevented participants from not receiving any treatment during the trial and also prevented the use of incorrect data in statistical analyses. These trials provide case studies for how health scientists can build monitoring systems for their digital intervention. Without these algorithm monitoring systems, critical issues would have gone undetected and unresolved. Instead, these monitoring systems safeguarded participants and ensured the quality of the resulting data for updating the intervention and facilitating scientific discovery. These monitoring guidelines and findings give digital intervention teams the confidence to include online decision-making algorithms in digital interventions. |
| title | Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2409.10526 |