Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.17073 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910915965747200 |
|---|---|
| author | Naeem, Awais Li, Tianhao Liao, Huang-Ru Xu, Jiawei Mathew, Aby M. Zhu, Zehao Tan, Zhen Jaiswal, Ajay Kumar Salibian, Raffi A. Hu, Ziniu Chen, Tianlong Ding, Ying |
| author_facet | Naeem, Awais Li, Tianhao Liao, Huang-Ru Xu, Jiawei Mathew, Aby M. Zhu, Zehao Tan, Zhen Jaiswal, Ajay Kumar Salibian, Raffi A. Hu, Ziniu Chen, Tianlong Ding, Ying |
| contents | Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17073 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering Naeem, Awais Li, Tianhao Liao, Huang-Ru Xu, Jiawei Mathew, Aby M. Zhu, Zehao Tan, Zhen Jaiswal, Ajay Kumar Salibian, Raffi A. Hu, Ziniu Chen, Tianlong Ding, Ying Computer Vision and Pattern Recognition Artificial Intelligence Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag). |
| title | Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2411.17073 |