Differentiate ChatGPT-generated and Human-written Medical Texts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909056372834304 |
|---|---|
| author | Liao, Wenxiong Liu, Zhengliang Dai, Haixing Xu, Shaochen Wu, Zihao Zhang, Yiyang Huang, Xiaoke Zhu, Dajiang Cai, Hongmin Liu, Tianming Li, Xiang |
| author_facet | Liao, Wenxiong Liu, Zhengliang Dai, Haixing Xu, Shaochen Wu, Zihao Zhang, Yiyang Huang, Xiaoke Zhu, Dajiang Cai, Hongmin Liu, Tianming Li, Xiang |
| contents | Background: Large language models such as ChatGPT are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the Internet. However, medical texts such as clinical notes and diagnoses require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to healthcare and the general public.
Objective: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT, and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT.
Methods: We first construct a suite of datasets containing medical texts written by human experts and generated by ChatGPT. In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally, we design and implement machine learning methods to detect medical text generated by ChatGPT.
Results: Medical texts written by humans are more concrete, more diverse, and typically contain more useful information, while medical texts generated by ChatGPT pay more attention to fluency and logic, and usually express general terminologies rather than effective information specific to the context of the problem. A BERT-based model can effectively detect medical texts generated by ChatGPT, and the F1 exceeds 95%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_11567 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Differentiate ChatGPT-generated and Human-written Medical Texts Liao, Wenxiong Liu, Zhengliang Dai, Haixing Xu, Shaochen Wu, Zihao Zhang, Yiyang Huang, Xiaoke Zhu, Dajiang Cai, Hongmin Liu, Tianming Li, Xiang Computation and Language Artificial Intelligence Background: Large language models such as ChatGPT are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the Internet. However, medical texts such as clinical notes and diagnoses require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to healthcare and the general public. Objective: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT, and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first construct a suite of datasets containing medical texts written by human experts and generated by ChatGPT. In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally, we design and implement machine learning methods to detect medical text generated by ChatGPT. Results: Medical texts written by humans are more concrete, more diverse, and typically contain more useful information, while medical texts generated by ChatGPT pay more attention to fluency and logic, and usually express general terminologies rather than effective information specific to the context of the problem. A BERT-based model can effectively detect medical texts generated by ChatGPT, and the F1 exceeds 95%. |
| title | Differentiate ChatGPT-generated and Human-written Medical Texts |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2304.11567 |