Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929616721018880 |
|---|---|
| author | Bucciarelli, Davide Moratelli, Nicholas Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita |
| author_facet | Bucciarelli, Davide Moratelli, Nicholas Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita |
| contents | The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03665 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis Bucciarelli, Davide Moratelli, Nicholas Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs. |
| title | Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia |
| url | https://arxiv.org/abs/2412.03665 |