Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929729695645696 |
|---|---|
| author | Liu, Che Wan, Zhongwei Wang, Yuqi Shen, Hui Wang, Haozhe Zheng, Kangyu Zhang, Mi Arcucci, Rossella |
| author_facet | Liu, Che Wan, Zhongwei Wang, Yuqi Shen, Hui Wang, Haozhe Zheng, Kangyu Zhang, Mi Arcucci, Rossella |
| contents | Automatic radiology report generation holds significant potential to streamline the labor-intensive process of report writing by radiologists, particularly for 3D radiographs such as CT scans. While CT scans are critical for clinical diagnostics, they remain less explored compared to 2D radiographs. To date, there has been no comprehensive benchmark for 3D radiograph report generation (3DRRG), nor sufficient investigation into the optimal training strategies for Vision Language Models (VLMs) in this context, particularly with respect to vision encoder choices, visual token compression, and model scaling. In this work, we make three key contributions. We curate **CT-3DRRG**, the largest **publicly** available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG. Furthermore, we propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data & model scale. Guided by these findings, we introduce **Argus**, a state-of-the-art family of VLMs that achieve superior performance across different model sizes and input 3D medical image resolutions, efficiently processing high-resolution 3D images up to $512 \times 512 \times 256$[^1]. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07146 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation Liu, Che Wan, Zhongwei Wang, Yuqi Shen, Hui Wang, Haozhe Zheng, Kangyu Zhang, Mi Arcucci, Rossella Computer Vision and Pattern Recognition Artificial Intelligence Automatic radiology report generation holds significant potential to streamline the labor-intensive process of report writing by radiologists, particularly for 3D radiographs such as CT scans. While CT scans are critical for clinical diagnostics, they remain less explored compared to 2D radiographs. To date, there has been no comprehensive benchmark for 3D radiograph report generation (3DRRG), nor sufficient investigation into the optimal training strategies for Vision Language Models (VLMs) in this context, particularly with respect to vision encoder choices, visual token compression, and model scaling. In this work, we make three key contributions. We curate **CT-3DRRG**, the largest **publicly** available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG. Furthermore, we propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data & model scale. Guided by these findings, we introduce **Argus**, a state-of-the-art family of VLMs that achieve superior performance across different model sizes and input 3D medical image resolutions, efficiently processing high-resolution 3D images up to $512 \times 512 \times 256$[^1]. |
| title | Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2406.07146 |