Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917693892853760 |
|---|---|
| author | Ma, Chong Jiang, Hanqi Chen, Wenting Li, Yiwei Wu, Zihao Yu, Xiaowei Liu, Zhengliang Guo, Lei Zhu, Dajiang Zhang, Tuo Shen, Dinggang Liu, Tianming Li, Xiang |
| author_facet | Ma, Chong Jiang, Hanqi Chen, Wenting Li, Yiwei Wu, Zihao Yu, Xiaowei Liu, Zhengliang Guo, Lei Zhu, Dajiang Zhang, Tuo Shen, Dinggang Liu, Tianming Li, Xiang |
| contents | In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12416 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning Ma, Chong Jiang, Hanqi Chen, Wenting Li, Yiwei Wu, Zihao Yu, Xiaowei Liu, Zhengliang Guo, Lei Zhu, Dajiang Zhang, Tuo Shen, Dinggang Liu, Tianming Li, Xiang Computer Vision and Pattern Recognition Computation and Language 68T07 I.2.0; I.4.0; I.5.4; I.7.0 In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework. |
| title | Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning |
| topic | Computer Vision and Pattern Recognition Computation and Language 68T07 I.2.0; I.4.0; I.5.4; I.7.0 |
| url | https://arxiv.org/abs/2403.12416 |