Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.10836 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916739472687104 |
|---|---|
| author | Dey, Abhishek Bothera, Aabha Sarikonda, Samhita Aryan, Rishav Podishetty, Sanjay Kumar Havalgi, Akshay Singh, Gaurav Srivastava, Saurabh |
| author_facet | Dey, Abhishek Bothera, Aabha Sarikonda, Samhita Aryan, Rishav Podishetty, Sanjay Kumar Havalgi, Akshay Singh, Gaurav Srivastava, Saurabh |
| contents | In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10836 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs Dey, Abhishek Bothera, Aabha Sarikonda, Samhita Aryan, Rishav Podishetty, Sanjay Kumar Havalgi, Akshay Singh, Gaurav Srivastava, Saurabh Computation and Language Computer Vision and Pattern Recognition In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches. |
| title | Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.10836 |