Iterative Prompt Refinement for Radiation Oncology Symptom Extraction Using Teacher-Student Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913224967847936 |
|---|---|
| author | Khanmohammadi, Reza Ghanem, Ahmed I Verdecchia, Kyle Hall, Ryan Elshaikh, Mohamed Movsas, Benjamin Bagher-Ebadian, Hassan Chetty, Indrin Ghassemi, Mohammad M. Thind, Kundan |
| author_facet | Khanmohammadi, Reza Ghanem, Ahmed I Verdecchia, Kyle Hall, Ryan Elshaikh, Mohamed Movsas, Benjamin Bagher-Ebadian, Hassan Chetty, Indrin Ghassemi, Mohammad M. Thind, Kundan |
| contents | This study introduces a novel teacher-student architecture utilizing Large Language Models (LLMs) to improve prostate cancer radiotherapy symptom extraction from clinical notes. Mixtral, the student model, initially extracts symptoms, followed by GPT-4, the teacher model, which refines prompts based on Mixtral's performance. This iterative process involved 294 single symptom clinical notes across 12 symptoms, with up to 16 rounds of refinement per epoch. Results showed significant improvements in extracting symptoms from both single and multi-symptom notes. For 59 single symptom notes, accuracy increased from 0.51 to 0.71, precision from 0.52 to 0.82, recall from 0.52 to 0.72, and F1 score from 0.49 to 0.73. In 375 multi-symptom notes, accuracy rose from 0.24 to 0.43, precision from 0.6 to 0.76, recall from 0.24 to 0.43, and F1 score from 0.20 to 0.44. These results demonstrate the effectiveness of advanced prompt engineering in LLMs for radiation oncology use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_04075 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Iterative Prompt Refinement for Radiation Oncology Symptom Extraction Using Teacher-Student Large Language Models Khanmohammadi, Reza Ghanem, Ahmed I Verdecchia, Kyle Hall, Ryan Elshaikh, Mohamed Movsas, Benjamin Bagher-Ebadian, Hassan Chetty, Indrin Ghassemi, Mohammad M. Thind, Kundan Computation and Language This study introduces a novel teacher-student architecture utilizing Large Language Models (LLMs) to improve prostate cancer radiotherapy symptom extraction from clinical notes. Mixtral, the student model, initially extracts symptoms, followed by GPT-4, the teacher model, which refines prompts based on Mixtral's performance. This iterative process involved 294 single symptom clinical notes across 12 symptoms, with up to 16 rounds of refinement per epoch. Results showed significant improvements in extracting symptoms from both single and multi-symptom notes. For 59 single symptom notes, accuracy increased from 0.51 to 0.71, precision from 0.52 to 0.82, recall from 0.52 to 0.72, and F1 score from 0.49 to 0.73. In 375 multi-symptom notes, accuracy rose from 0.24 to 0.43, precision from 0.6 to 0.76, recall from 0.24 to 0.43, and F1 score from 0.20 to 0.44. These results demonstrate the effectiveness of advanced prompt engineering in LLMs for radiation oncology use. |
| title | Iterative Prompt Refinement for Radiation Oncology Symptom Extraction Using Teacher-Student Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.04075 |