A Clinical Trial Design Approach to Auditing Language Models in Healthcare Setting
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916532141948928 |
|---|---|
| author | Gondara, Lovedeep Simkin, Jonathan |
| author_facet | Gondara, Lovedeep Simkin, Jonathan |
| contents | We present an audit mechanism for language models, with a focus on models deployed in the healthcare setting. Our proposed mechanism takes inspiration from clinical trial design where we posit the language model audit as a single blind equivalence trial, with the comparison of interest being the subject matter experts. We show that using our proposed method, we can follow principled sample size and power calculations, leading to the requirement of sampling minimum number of records while maintaining the audit integrity and statistical soundness. Finally, we provide a real-world example of the audit used in a production environment in a large-scale public health network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16702 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Clinical Trial Design Approach to Auditing Language Models in Healthcare Setting Gondara, Lovedeep Simkin, Jonathan Computers and Society Machine Learning We present an audit mechanism for language models, with a focus on models deployed in the healthcare setting. Our proposed mechanism takes inspiration from clinical trial design where we posit the language model audit as a single blind equivalence trial, with the comparison of interest being the subject matter experts. We show that using our proposed method, we can follow principled sample size and power calculations, leading to the requirement of sampling minimum number of records while maintaining the audit integrity and statistical soundness. Finally, we provide a real-world example of the audit used in a production environment in a large-scale public health network. |
| title | A Clinical Trial Design Approach to Auditing Language Models in Healthcare Setting |
| topic | Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2411.16702 |