A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913398439018496 |
|---|---|
| author | Kaliosis, Panagiotis Pavlopoulos, John Charalampakos, Foivos Moschovis, Georgios Androutsopoulos, Ion |
| author_facet | Kaliosis, Panagiotis Pavlopoulos, John Charalampakos, Foivos Moschovis, Georgios Androutsopoulos, Ion |
| contents | Diagnostic Captioning (DC) automatically generates a diagnostic text from one or more medical images (e.g., X-rays, MRIs) of a patient. Treated as a draft, the generated text may assist clinicians, by providing an initial estimation of the patient's condition, speeding up and helping safeguard the diagnostic process. The accuracy of a diagnostic text, however, strongly depends on how well the key medical conditions depicted in the images are expressed. We propose a new data-driven guided decoding method that incorporates medical information, in the form of existing tags capturing key conditions of the image(s), into the beam search of the diagnostic text generation process. We evaluate the proposed method on two medical datasets using four DC systems that range from generic image-to-text systems with CNN encoders and RNN decoders to pre-trained Large Language Models. The latter can also be used in few- and zero-shot learning scenarios. In most cases, the proposed mechanism improves performance with respect to all evaluation measures. We provide an open-source implementation of the proposed method at https://github.com/nlpaueb/dmmcs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14164 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning Kaliosis, Panagiotis Pavlopoulos, John Charalampakos, Foivos Moschovis, Georgios Androutsopoulos, Ion Artificial Intelligence Computation and Language Diagnostic Captioning (DC) automatically generates a diagnostic text from one or more medical images (e.g., X-rays, MRIs) of a patient. Treated as a draft, the generated text may assist clinicians, by providing an initial estimation of the patient's condition, speeding up and helping safeguard the diagnostic process. The accuracy of a diagnostic text, however, strongly depends on how well the key medical conditions depicted in the images are expressed. We propose a new data-driven guided decoding method that incorporates medical information, in the form of existing tags capturing key conditions of the image(s), into the beam search of the diagnostic text generation process. We evaluate the proposed method on two medical datasets using four DC systems that range from generic image-to-text systems with CNN encoders and RNN decoders to pre-trained Large Language Models. The latter can also be used in few- and zero-shot learning scenarios. In most cases, the proposed mechanism improves performance with respect to all evaluation measures. We provide an open-source implementation of the proposed method at https://github.com/nlpaueb/dmmcs. |
| title | A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.14164 |