Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916912807542784 |
|---|---|
| author | Huy, Ta Duc Huynh, Duy Anh Xie, Yutong Qi, Yuankai Chen, Qi Nguyen, Phi Le Tran, Sen Kim Phung, Son Lam Hengel, Anton van den Liao, Zhibin To, Minh-Son Verjans, Johan W. Phan, Vu Minh Hieu |
| author_facet | Huy, Ta Duc Huynh, Duy Anh Xie, Yutong Qi, Yuankai Chen, Qi Nguyen, Phi Le Tran, Sen Kim Phung, Son Lam Hengel, Anton van den Liao, Zhibin To, Minh-Son Verjans, Johan W. Phan, Vu Minh Hieu |
| contents | Visual grounding (VG) is the capability to identify the specific regions in an image associated with a particular text description. In medical imaging, VG enhances interpretability by highlighting relevant pathological features corresponding to textual descriptions, improving model transparency and trustworthiness for wider adoption of deep learning models in clinical practice. Current models struggle to associate textual descriptions with disease regions due to inefficient attention mechanisms and a lack of fine-grained token representations. In this paper, we empirically demonstrate two key observations. First, current VLMs assign high norms to background tokens, diverting the model's attention from regions of disease. Second, the global tokens used for cross-modal learning are not representative of local disease tokens. This hampers identifying correlations between the text and disease tokens. To address this, we introduce simple, yet effective Disease-Aware Prompting (DAP) process, which uses the explainability map of a VLM to identify the appropriate image features. This simple strategy amplifies disease-relevant regions while suppressing background interference. Without any additional pixel-level annotations, DAP improves visual grounding accuracy by 20.74% compared to state-of-the-art methods across three major chest X-ray datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15123 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding Huy, Ta Duc Huynh, Duy Anh Xie, Yutong Qi, Yuankai Chen, Qi Nguyen, Phi Le Tran, Sen Kim Phung, Son Lam Hengel, Anton van den Liao, Zhibin To, Minh-Son Verjans, Johan W. Phan, Vu Minh Hieu Computer Vision and Pattern Recognition Artificial Intelligence Visual grounding (VG) is the capability to identify the specific regions in an image associated with a particular text description. In medical imaging, VG enhances interpretability by highlighting relevant pathological features corresponding to textual descriptions, improving model transparency and trustworthiness for wider adoption of deep learning models in clinical practice. Current models struggle to associate textual descriptions with disease regions due to inefficient attention mechanisms and a lack of fine-grained token representations. In this paper, we empirically demonstrate two key observations. First, current VLMs assign high norms to background tokens, diverting the model's attention from regions of disease. Second, the global tokens used for cross-modal learning are not representative of local disease tokens. This hampers identifying correlations between the text and disease tokens. To address this, we introduce simple, yet effective Disease-Aware Prompting (DAP) process, which uses the explainability map of a VLM to identify the appropriate image features. This simple strategy amplifies disease-relevant regions while suppressing background interference. Without any additional pixel-level annotations, DAP improves visual grounding accuracy by 20.74% compared to state-of-the-art methods across three major chest X-ray datasets. |
| title | Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2505.15123 |