PneumoLLM: Harnessing the Power of Large Language Model for Pneumoconiosis Diagnosis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913408611254272 |
|---|---|
| author | Song, Meiyue Yu, Zhihua Wang, Jiaxin Wang, Jiarui Lu, Yuting Li, Baicun Wang, Xiaoxu Huang, Qinghua Li, Zhijun Kanellakis, Nikolaos I. Liu, Jiangfeng Wang, Jing Wang, Binglu Yang, Juntao |
| author_facet | Song, Meiyue Yu, Zhihua Wang, Jiaxin Wang, Jiarui Lu, Yuting Li, Baicun Wang, Xiaoxu Huang, Qinghua Li, Zhijun Kanellakis, Nikolaos I. Liu, Jiangfeng Wang, Jing Wang, Binglu Yang, Juntao |
| contents | The conventional pretraining-and-finetuning paradigm, while effective for common diseases with ample data, faces challenges in diagnosing data-scarce occupational diseases like pneumoconiosis. Recently, large language models (LLMs) have exhibits unprecedented ability when conducting multiple tasks in dialogue, bringing opportunities to diagnosis. A common strategy might involve using adapter layers for vision-language alignment and diagnosis in a dialogic manner. Yet, this approach often requires optimization of extensive learnable parameters in the text branch and the dialogue head, potentially diminishing the LLMs' efficacy, especially with limited training data. In our work, we innovate by eliminating the text branch and substituting the dialogue head with a classification head. This approach presents a more effective method for harnessing LLMs in diagnosis with fewer learnable parameters. Furthermore, to balance the retention of detailed image information with progression towards accurate diagnosis, we introduce the contextual multi-token engine. This engine is specialized in adaptively generating diagnostic tokens. Additionally, we propose the information emitter module, which unidirectionally emits information from image tokens to diagnosis tokens. Comprehensive experiments validate the superiority of our methods and the effectiveness of proposed modules. Our codes can be found at https://github.com/CodeMonsterPHD/PneumoLLM/tree/main. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_03490 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PneumoLLM: Harnessing the Power of Large Language Model for Pneumoconiosis Diagnosis Song, Meiyue Yu, Zhihua Wang, Jiaxin Wang, Jiarui Lu, Yuting Li, Baicun Wang, Xiaoxu Huang, Qinghua Li, Zhijun Kanellakis, Nikolaos I. Liu, Jiangfeng Wang, Jing Wang, Binglu Yang, Juntao Image and Video Processing Computer Vision and Pattern Recognition The conventional pretraining-and-finetuning paradigm, while effective for common diseases with ample data, faces challenges in diagnosing data-scarce occupational diseases like pneumoconiosis. Recently, large language models (LLMs) have exhibits unprecedented ability when conducting multiple tasks in dialogue, bringing opportunities to diagnosis. A common strategy might involve using adapter layers for vision-language alignment and diagnosis in a dialogic manner. Yet, this approach often requires optimization of extensive learnable parameters in the text branch and the dialogue head, potentially diminishing the LLMs' efficacy, especially with limited training data. In our work, we innovate by eliminating the text branch and substituting the dialogue head with a classification head. This approach presents a more effective method for harnessing LLMs in diagnosis with fewer learnable parameters. Furthermore, to balance the retention of detailed image information with progression towards accurate diagnosis, we introduce the contextual multi-token engine. This engine is specialized in adaptively generating diagnostic tokens. Additionally, we propose the information emitter module, which unidirectionally emits information from image tokens to diagnosis tokens. Comprehensive experiments validate the superiority of our methods and the effectiveness of proposed modules. Our codes can be found at https://github.com/CodeMonsterPHD/PneumoLLM/tree/main. |
| title | PneumoLLM: Harnessing the Power of Large Language Model for Pneumoconiosis Diagnosis |
| topic | Image and Video Processing Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.03490 |