Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917482547118080 |
|---|---|
| author | Seo, Jean Kim, Gibaeg Shin, Kihun Lim, Seungseop Lee, Hyunkyung Han, Wooseok Lee, Jongwon Yang, Eunho |
| author_facet | Seo, Jean Kim, Gibaeg Shin, Kihun Lim, Seungseop Lee, Hyunkyung Han, Wooseok Lee, Jongwon Yang, Eunho |
| contents | We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03627 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines Seo, Jean Kim, Gibaeg Shin, Kihun Lim, Seungseop Lee, Hyunkyung Han, Wooseok Lee, Jongwon Yang, Eunho Computation and Language Artificial Intelligence We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings. |
| title | Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.03627 |