Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.00072 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909630206050304 |
|---|---|
| author | Naderi, Nariman Atf, Zahra Lewis, Peter R far, Aref Mahjoub Safavi-Naini, Seyed Amir Ahmad Soroush, Ali |
| author_facet | Naderi, Nariman Atf, Zahra Lewis, Peter R far, Aref Mahjoub Safavi-Naini, Seyed Amir Ahmad Soroush, Ali |
| contents | This paper investigates how prompt engineering techniques impact both accuracy and confidence elicitation in Large Language Models (LLMs) applied to medical contexts. Using a stratified dataset of Persian board exam questions across multiple specialties, we evaluated five LLMs - GPT-4o, o3-mini, Llama-3.3-70b, Llama-3.1-8b, and DeepSeek-v3 - across 156 configurations. These configurations varied in temperature settings (0.3, 0.7, 1.0), prompt styles (Chain-of-Thought, Few-Shot, Emotional, Expert Mimicry), and confidence scales (1-10, 1-100). We used AUC-ROC, Brier Score, and Expected Calibration Error (ECE) to evaluate alignment between confidence and actual performance. Chain-of-Thought prompts improved accuracy but also led to overconfidence, highlighting the need for calibration. Emotional prompting further inflated confidence, risking poor decisions. Smaller models like Llama-3.1-8b underperformed across all metrics, while proprietary models showed higher accuracy but still lacked calibrated confidence. These results suggest prompt engineering must address both accuracy and uncertainty to be effective in high-stakes medical tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00072 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs Naderi, Nariman Atf, Zahra Lewis, Peter R far, Aref Mahjoub Safavi-Naini, Seyed Amir Ahmad Soroush, Ali Computers and Society Artificial Intelligence Computation and Language Machine Learning This paper investigates how prompt engineering techniques impact both accuracy and confidence elicitation in Large Language Models (LLMs) applied to medical contexts. Using a stratified dataset of Persian board exam questions across multiple specialties, we evaluated five LLMs - GPT-4o, o3-mini, Llama-3.3-70b, Llama-3.1-8b, and DeepSeek-v3 - across 156 configurations. These configurations varied in temperature settings (0.3, 0.7, 1.0), prompt styles (Chain-of-Thought, Few-Shot, Emotional, Expert Mimicry), and confidence scales (1-10, 1-100). We used AUC-ROC, Brier Score, and Expected Calibration Error (ECE) to evaluate alignment between confidence and actual performance. Chain-of-Thought prompts improved accuracy but also led to overconfidence, highlighting the need for calibration. Emotional prompting further inflated confidence, risking poor decisions. Smaller models like Llama-3.1-8b underperformed across all metrics, while proprietary models showed higher accuracy but still lacked calibrated confidence. These results suggest prompt engineering must address both accuracy and uncertainty to be effective in high-stakes medical tasks. |
| title | Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs |
| topic | Computers and Society Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.00072 |