Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929713353588736 |
|---|---|
| author | Nissen, Leon Zagar, Philipp Ravi, Vishnu Zahedivash, Aydin Reimer, Lara Marie Jonas, Stephan Aalami, Oliver Schmiedmayer, Paul |
| author_facet | Nissen, Leon Zagar, Philipp Ravi, Vishnu Zahedivash, Aydin Reimer, Lara Marie Jonas, Stephan Aalami, Oliver Schmiedmayer, Paul |
| contents | The deployment of Large Language Models (LLM) on mobile devices offers significant potential for medical applications, enhancing privacy, security, and cost-efficiency by eliminating reliance on cloud-based services and keeping sensitive health data local. However, the performance and accuracy of on-device LLMs in real-world medical contexts remain underexplored. In this study, we benchmark publicly available on-device LLMs using the AMEGA dataset, evaluating accuracy, computational efficiency, and thermal limitation across various mobile devices. Our results indicate that compact general-purpose models like Phi-3 Mini achieve a strong balance between speed and accuracy, while medically fine-tuned models such as Med42 and Aloe attain the highest accuracy. Notably, deploying LLMs on older devices remains feasible, with memory constraints posing a greater challenge than raw processing power. Our study underscores the potential of on-device LLMs for healthcare while emphasizing the need for more efficient inference and models tailored to real-world clinical reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08954 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning Nissen, Leon Zagar, Philipp Ravi, Vishnu Zahedivash, Aydin Reimer, Lara Marie Jonas, Stephan Aalami, Oliver Schmiedmayer, Paul Computation and Language The deployment of Large Language Models (LLM) on mobile devices offers significant potential for medical applications, enhancing privacy, security, and cost-efficiency by eliminating reliance on cloud-based services and keeping sensitive health data local. However, the performance and accuracy of on-device LLMs in real-world medical contexts remain underexplored. In this study, we benchmark publicly available on-device LLMs using the AMEGA dataset, evaluating accuracy, computational efficiency, and thermal limitation across various mobile devices. Our results indicate that compact general-purpose models like Phi-3 Mini achieve a strong balance between speed and accuracy, while medically fine-tuned models such as Med42 and Aloe attain the highest accuracy. Notably, deploying LLMs on older devices remains feasible, with memory constraints posing a greater challenge than raw processing power. Our study underscores the potential of on-device LLMs for healthcare while emphasizing the need for more efficient inference and models tailored to real-world clinical reasoning. |
| title | Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.08954 |