Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.03627 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917482547118080 |
|---|---|
| author | Seo, Jean Kim, Gibaeg Shin, Kihun Lim, Seungseop Lee, Hyunkyung Han, Wooseok Lee, Jongwon Yang, Eunho |
| author_facet | Seo, Jean Kim, Gibaeg Shin, Kihun Lim, Seungseop Lee, Hyunkyung Han, Wooseok Lee, Jongwon Yang, Eunho |
| contents | We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03627 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines Seo, Jean Kim, Gibaeg Shin, Kihun Lim, Seungseop Lee, Hyunkyung Han, Wooseok Lee, Jongwon Yang, Eunho Computation and Language Artificial Intelligence We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings. |
| title | Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.03627 |