Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916739472687104 |
|---|---|
| author | Dey, Abhishek Bothera, Aabha Sarikonda, Samhita Aryan, Rishav Podishetty, Sanjay Kumar Havalgi, Akshay Singh, Gaurav Srivastava, Saurabh |
| author_facet | Dey, Abhishek Bothera, Aabha Sarikonda, Samhita Aryan, Rishav Podishetty, Sanjay Kumar Havalgi, Akshay Singh, Gaurav Srivastava, Saurabh |
| contents | In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10836 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs Dey, Abhishek Bothera, Aabha Sarikonda, Samhita Aryan, Rishav Podishetty, Sanjay Kumar Havalgi, Akshay Singh, Gaurav Srivastava, Saurabh Computation and Language Computer Vision and Pattern Recognition In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches. |
| title | Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.10836 |