Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911938485682176 |
|---|---|
| author | Hsu, Chih-Chung Lee, Chia-Ming Lin, Yu-Fan Chou, Yi-Shiuan Jian, Chih-Yu Tsai, Chi-Han |
| author_facet | Hsu, Chih-Chung Lee, Chia-Ming Lin, Yu-Fan Chou, Yi-Shiuan Jian, Chih-Yu Tsai, Chi-Han |
| contents | Social media popularity (SMP) prediction is a complex task involving multi-modal data integration. While pre-trained vision-language models (VLMs) like CLIP have been widely adopted for this task, their effectiveness in capturing the unique characteristics of social media content remains unexplored. This paper critically examines the applicability of CLIP-based features in SMP prediction, focusing on the overlooked phenomenon of semantic inconsistency between images and text in social media posts. Through extensive analysis, we demonstrate that this inconsistency increases with post popularity, challenging the conventional use of VLM features. We provide a comprehensive investigation of semantic inconsistency across different popularity intervals and analyze the impact of VLM feature adaptation on SMP tasks. Our experiments reveal that incorporating inconsistency measures and adapted text features significantly improves model performance, achieving an SRC of 0.729 and an MAE of 1.227. These findings not only enhance SMP prediction accuracy but also provide crucial insights for developing more targeted approaches in social media analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00556 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction Hsu, Chih-Chung Lee, Chia-Ming Lin, Yu-Fan Chou, Yi-Shiuan Jian, Chih-Yu Tsai, Chi-Han Multimedia Social media popularity (SMP) prediction is a complex task involving multi-modal data integration. While pre-trained vision-language models (VLMs) like CLIP have been widely adopted for this task, their effectiveness in capturing the unique characteristics of social media content remains unexplored. This paper critically examines the applicability of CLIP-based features in SMP prediction, focusing on the overlooked phenomenon of semantic inconsistency between images and text in social media posts. Through extensive analysis, we demonstrate that this inconsistency increases with post popularity, challenging the conventional use of VLM features. We provide a comprehensive investigation of semantic inconsistency across different popularity intervals and analyze the impact of VLM feature adaptation on SMP tasks. Our experiments reveal that incorporating inconsistency measures and adapted text features significantly improves model performance, achieving an SRC of 0.729 and an MAE of 1.227. These findings not only enhance SMP prediction accuracy but also provide crucial insights for developing more targeted approaches in social media analysis. |
| title | Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction |
| topic | Multimedia |
| url | https://arxiv.org/abs/2407.00556 |