SoMeLVLM: A Large Vision Language Model for Social Media Processing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910643004637184 |
|---|---|
| author | Zhang, Xinnong Kuang, Haoyu Mou, Xinyi Lyu, Hanjia Wu, Kun Chen, Siming Luo, Jiebo Huang, Xuanjing Wei, Zhongyu |
| author_facet | Zhang, Xinnong Kuang, Haoyu Mou, Xinyi Lyu, Hanjia Wu, Kun Chen, Siming Luo, Jiebo Huang, Xuanjing Wei, Zhongyu |
| contents | The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13022 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SoMeLVLM: A Large Vision Language Model for Social Media Processing Zhang, Xinnong Kuang, Haoyu Mou, Xinyi Lyu, Hanjia Wu, Kun Chen, Siming Luo, Jiebo Huang, Xuanjing Wei, Zhongyu Computation and Language Multimedia The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities. |
| title | SoMeLVLM: A Large Vision Language Model for Social Media Processing |
| topic | Computation and Language Multimedia |
| url | https://arxiv.org/abs/2402.13022 |