Improving Medical Visual Representations via Radiology Report Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912182411722752 |
|---|---|
| author | Quigley, Keegan Cha, Miriam Barua, Josh Chauhan, Geeticka Berkowitz, Seth Horng, Steven Golland, Polina |
| author_facet | Quigley, Keegan Cha, Miriam Barua, Josh Chauhan, Geeticka Berkowitz, Seth Horng, Steven Golland, Polina |
| contents | Vision-language pretraining has been shown to produce high-quality visual encoders which transfer efficiently to downstream computer vision tasks. Contrastive learning approaches have increasingly been adopted for medical vision language pretraining (MVLP), yet recent developments in generative AI offer new modeling alternatives. This paper introduces RadTex, a CNN-encoder transformer-decoder architecture optimized for radiology. We explore bidirectional captioning as an alternative MVLP strategy and demonstrate that RadTex's captioning pretraining is competitive with established contrastive methods, achieving a CheXpert macro-AUC of 89.4%. Additionally, RadTex's lightweight text decoder not only generates clinically relevant radiology reports (macro-F1 score of 0.349), but also provides targeted, interactive responses, highlighting the utility of bidirectional captioning in advancing medical image analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_19635 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Improving Medical Visual Representations via Radiology Report Generation Quigley, Keegan Cha, Miriam Barua, Josh Chauhan, Geeticka Berkowitz, Seth Horng, Steven Golland, Polina Computer Vision and Pattern Recognition Vision-language pretraining has been shown to produce high-quality visual encoders which transfer efficiently to downstream computer vision tasks. Contrastive learning approaches have increasingly been adopted for medical vision language pretraining (MVLP), yet recent developments in generative AI offer new modeling alternatives. This paper introduces RadTex, a CNN-encoder transformer-decoder architecture optimized for radiology. We explore bidirectional captioning as an alternative MVLP strategy and demonstrate that RadTex's captioning pretraining is competitive with established contrastive methods, achieving a CheXpert macro-AUC of 89.4%. Additionally, RadTex's lightweight text decoder not only generates clinically relevant radiology reports (macro-F1 score of 0.349), but also provides targeted, interactive responses, highlighting the utility of bidirectional captioning in advancing medical image analysis. |
| title | Improving Medical Visual Representations via Radiology Report Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2310.19635 |