Inserting Faces inside Captions: Image Captioning with Attention Guided Merging
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911866190561280 |
|---|---|
| author | Tevissen, Yannis Guetari, Khalil Tassel, Marine Kerleroux, Erwan Petitpont, Frédéric |
| author_facet | Tevissen, Yannis Guetari, Khalil Tassel, Marine Kerleroux, Erwan Petitpont, Frédéric |
| contents | Image captioning models are widely used to describe recent and archived pictures with the objective of improving their accessibility and retrieval. Yet, these approaches tend to be inefficient and biased at retrieving people's names. In this work we introduce AstroCaptions, a dataset for the image captioning task. This dataset specifically contains thousands of public fig-ures that are complex to identify for a traditional model. We also propose a novel post-processing method to insert identified people's names inside the caption using explainable AI tools and the grounding capabilities of vi-sion-language models. The results obtained with this method show signifi-cant improvements of captions quality and a potential of reducing halluci-nations. Up to 93.2% of the persons detected can be inserted in the image captions leading to improvements in the BLEU, ROUGE, CIDEr and METEOR scores of each captioning model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_02305 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inserting Faces inside Captions: Image Captioning with Attention Guided Merging Tevissen, Yannis Guetari, Khalil Tassel, Marine Kerleroux, Erwan Petitpont, Frédéric Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Image and Video Processing Image captioning models are widely used to describe recent and archived pictures with the objective of improving their accessibility and retrieval. Yet, these approaches tend to be inefficient and biased at retrieving people's names. In this work we introduce AstroCaptions, a dataset for the image captioning task. This dataset specifically contains thousands of public fig-ures that are complex to identify for a traditional model. We also propose a novel post-processing method to insert identified people's names inside the caption using explainable AI tools and the grounding capabilities of vi-sion-language models. The results obtained with this method show signifi-cant improvements of captions quality and a potential of reducing halluci-nations. Up to 93.2% of the persons detected can be inserted in the image captions leading to improvements in the BLEU, ROUGE, CIDEr and METEOR scores of each captioning model. |
| title | Inserting Faces inside Captions: Image Captioning with Attention Guided Merging |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Image and Video Processing |
| url | https://arxiv.org/abs/2405.02305 |