Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2411.16702 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916532141948928 |
|---|---|
| author | Gondara, Lovedeep Simkin, Jonathan |
| author_facet | Gondara, Lovedeep Simkin, Jonathan |
| contents | We present an audit mechanism for language models, with a focus on models deployed in the healthcare setting. Our proposed mechanism takes inspiration from clinical trial design where we posit the language model audit as a single blind equivalence trial, with the comparison of interest being the subject matter experts. We show that using our proposed method, we can follow principled sample size and power calculations, leading to the requirement of sampling minimum number of records while maintaining the audit integrity and statistical soundness. Finally, we provide a real-world example of the audit used in a production environment in a large-scale public health network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16702 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Clinical Trial Design Approach to Auditing Language Models in Healthcare Setting Gondara, Lovedeep Simkin, Jonathan Computers and Society Machine Learning We present an audit mechanism for language models, with a focus on models deployed in the healthcare setting. Our proposed mechanism takes inspiration from clinical trial design where we posit the language model audit as a single blind equivalence trial, with the comparison of interest being the subject matter experts. We show that using our proposed method, we can follow principled sample size and power calculations, leading to the requirement of sampling minimum number of records while maintaining the audit integrity and statistical soundness. Finally, we provide a real-world example of the audit used in a production environment in a large-scale public health network. |
| title | A Clinical Trial Design Approach to Auditing Language Models in Healthcare Setting |
| topic | Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2411.16702 |