Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929354512007168 |
|---|---|
| author | Zhong, Zhusi Li, Jie Sollee, John Collins, Scott Bai, Harrison Zhang, Paul Healey, Terrence Atalay, Michael Gao, Xinbo Jiao, Zhicheng |
| author_facet | Zhong, Zhusi Li, Jie Sollee, John Collins, Scott Bai, Harrison Zhang, Paul Healey, Terrence Atalay, Michael Gao, Xinbo Jiao, Zhicheng |
| contents | In response to the worldwide COVID-19 pandemic, advanced automated technologies have emerged as valuable tools to aid healthcare professionals in managing an increased workload by improving radiology report generation and prognostic analysis. This study proposes Multi-modality Regional Alignment Network (MRANet), an explainable model for radiology report generation and survival prediction that focuses on high-risk regions. By learning spatial correlation in the detector, MRANet visually grounds region-specific descriptions, providing robust anatomical regions with a completion strategy. The visual features of each region are embedded using a novel survival attention mechanism, offering spatially and risk-aware features for sentence encoding while maintaining global coherence across tasks. A cross LLMs alignment is employed to enhance the image-to-text transfer process, resulting in sentences rich with clinical detail and improved explainability for radiologist. Multi-center experiments validate both MRANet's overall performance and each module's composition within the model, encouraging further advancements in radiology report generation research emphasizing clinical interpretation and trustworthiness in AI models applied to medical studies. The code is available at https://github.com/zzs95/MRANet. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14113 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation Zhong, Zhusi Li, Jie Sollee, John Collins, Scott Bai, Harrison Zhang, Paul Healey, Terrence Atalay, Michael Gao, Xinbo Jiao, Zhicheng Image and Video Processing Computer Vision and Pattern Recognition In response to the worldwide COVID-19 pandemic, advanced automated technologies have emerged as valuable tools to aid healthcare professionals in managing an increased workload by improving radiology report generation and prognostic analysis. This study proposes Multi-modality Regional Alignment Network (MRANet), an explainable model for radiology report generation and survival prediction that focuses on high-risk regions. By learning spatial correlation in the detector, MRANet visually grounds region-specific descriptions, providing robust anatomical regions with a completion strategy. The visual features of each region are embedded using a novel survival attention mechanism, offering spatially and risk-aware features for sentence encoding while maintaining global coherence across tasks. A cross LLMs alignment is employed to enhance the image-to-text transfer process, resulting in sentences rich with clinical detail and improved explainability for radiologist. Multi-center experiments validate both MRANet's overall performance and each module's composition within the model, encouraging further advancements in radiology report generation research emphasizing clinical interpretation and trustworthiness in AI models applied to medical studies. The code is available at https://github.com/zzs95/MRANet. |
| title | Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation |
| topic | Image and Video Processing Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.14113 |