Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.04221 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913092002119680 |
|---|---|
| author | Chuang, Yao-Shun Mody, Tushti Singh, Uday Pratap Shiraz, Shirindokht Lee, Chun-Teh Brandon, Ryan Walji, Muhammad F Jiang, Xiaoqian Tokede, Bunmi |
| author_facet | Chuang, Yao-Shun Mody, Tushti Singh, Uday Pratap Shiraz, Shirindokht Lee, Chun-Teh Brandon, Ryan Walji, Muhammad F Jiang, Xiaoqian Tokede, Bunmi |
| contents | Clinical named entity recognition from dental progress notes is challenging because documentation is highly unstructured, domain-specific, and often privacy-sensitive. We developed a locally deployable framework that enables small language models to self-generate, verify, refine, and evaluate entity-specific prompts for extracting multiple clinical entities from dental notes. Using 1,200 annotated notes, we evaluated candidate open-weight models with multi-prompt ensemble inference and further adapted selected models using QLoRA-based supervised fine-tuning and direct preference optimization. Model performance varied substantially, highlighting the need for task-specific evaluation rather than reliance on generic benchmarks. Qwen2.5-14B-Instruct achieved the strongest baseline performance. After DPO, Qwen2.5-14B-Instruct and Llama-3.1-8B-Instruct achieved micro/macro F1 scores of 0.864/0.837 and 0.806/0.797, respectively. These findings suggest that automated prompt optimization combined with lightweight preference-based post-training can support scalable clinical information extraction using locally deployed small language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04221 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction Chuang, Yao-Shun Mody, Tushti Singh, Uday Pratap Shiraz, Shirindokht Lee, Chun-Teh Brandon, Ryan Walji, Muhammad F Jiang, Xiaoqian Tokede, Bunmi Computation and Language Artificial Intelligence Clinical named entity recognition from dental progress notes is challenging because documentation is highly unstructured, domain-specific, and often privacy-sensitive. We developed a locally deployable framework that enables small language models to self-generate, verify, refine, and evaluate entity-specific prompts for extracting multiple clinical entities from dental notes. Using 1,200 annotated notes, we evaluated candidate open-weight models with multi-prompt ensemble inference and further adapted selected models using QLoRA-based supervised fine-tuning and direct preference optimization. Model performance varied substantially, highlighting the need for task-specific evaluation rather than reliance on generic benchmarks. Qwen2.5-14B-Instruct achieved the strongest baseline performance. After DPO, Qwen2.5-14B-Instruct and Llama-3.1-8B-Instruct achieved micro/macro F1 scores of 0.864/0.837 and 0.806/0.797, respectively. These findings suggest that automated prompt optimization combined with lightweight preference-based post-training can support scalable clinical information extraction using locally deployed small language models. |
| title | Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.04221 |