Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.20794 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908292344709120 |
|---|---|
| author | Kocaman, Veysel Santas, Muhammed Gul, Yigit Butgul, Mehmet Talby, David |
| author_facet | Kocaman, Veysel Santas, Muhammed Gul, Yigit Butgul, Mehmet Talby, David |
| contents | We evaluate the performance of four leading solutions for de-identification of unstructured medical text - Azure Health Data Services, AWS Comprehend Medical, OpenAI GPT-4o, and John Snow Labs - on a ground truth dataset of 48 clinical documents annotated by medical experts. The analysis, conducted at both entity-level and token-level, suggests that John Snow Labs' Medical Language Models solution achieves the highest accuracy, with a 96% F1-score in protected health information (PHI) detection, outperforming Azure (91%), AWS (83%), and GPT-4o (79%). John Snow Labs is not only the only solution which achieves regulatory-grade accuracy (surpassing that of human experts) but is also the most cost-effective solution: It is over 80% cheaper compared to Azure and GPT-4o, and is the only solution not priced by token. Its fixed-cost local deployment model avoids the escalating per-request fees of cloud-based services, making it a scalable and economical choice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_20794 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text DeIdentification? Kocaman, Veysel Santas, Muhammed Gul, Yigit Butgul, Mehmet Talby, David Computation and Language Cryptography and Security Information Retrieval Machine Learning H.3; F.2.2; I.2.7 We evaluate the performance of four leading solutions for de-identification of unstructured medical text - Azure Health Data Services, AWS Comprehend Medical, OpenAI GPT-4o, and John Snow Labs - on a ground truth dataset of 48 clinical documents annotated by medical experts. The analysis, conducted at both entity-level and token-level, suggests that John Snow Labs' Medical Language Models solution achieves the highest accuracy, with a 96% F1-score in protected health information (PHI) detection, outperforming Azure (91%), AWS (83%), and GPT-4o (79%). John Snow Labs is not only the only solution which achieves regulatory-grade accuracy (surpassing that of human experts) but is also the most cost-effective solution: It is over 80% cheaper compared to Azure and GPT-4o, and is the only solution not priced by token. Its fixed-cost local deployment model avoids the escalating per-request fees of cloud-based services, making it a scalable and economical choice. |
| title | Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text DeIdentification? |
| topic | Computation and Language Cryptography and Security Information Retrieval Machine Learning H.3; F.2.2; I.2.7 |
| url | https://arxiv.org/abs/2503.20794 |