Expanding Event Modality Applications through a Robust CLIP-Based Encoder
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909603882598400 |
|---|---|
| author | Jeong, Sungheon Chen, Hanning Yun, Sanggeon Cho, Suhyeon Huang, Wenjun Liu, Xiangjian Imani, Mohsen |
| author_facet | Jeong, Sungheon Chen, Hanning Yun, Sanggeon Cho, Suhyeon Huang, Wenjun Liu, Xiangjian Imani, Mohsen |
| contents | This paper introduces a powerful encoder that transfers CLIP`s capabilities to event-based data, enhancing its utility and expanding its applicability across diverse domains. While large-scale datasets have significantly advanced image-based models, the scarcity of comprehensive event datasets has limited performance potential in event modality. To address this challenge, we adapt CLIP`s architecture to align event embeddings with image embeddings, supporting zero-shot learning and preserving text alignment while mitigating catastrophic forgetting. Our encoder achieves strong performance in object recognition, with competitive results in zero-shot and few-shot learning tasks. Notably, it generalizes effectively to events extracted from video data without requiring additional training, highlighting its versatility. Additionally, we integrate this encoder within a cross-modality framework that facilitates interaction across five modalities-Image, Event, Text, Sound, and Depth-expanding the possibilities for cross-modal applications. Overall, this work underscores the transformative potential of a robust event encoder, broadening the scope and utility of event-based data across various fields. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03093 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Expanding Event Modality Applications through a Robust CLIP-Based Encoder Jeong, Sungheon Chen, Hanning Yun, Sanggeon Cho, Suhyeon Huang, Wenjun Liu, Xiangjian Imani, Mohsen Computer Vision and Pattern Recognition This paper introduces a powerful encoder that transfers CLIP`s capabilities to event-based data, enhancing its utility and expanding its applicability across diverse domains. While large-scale datasets have significantly advanced image-based models, the scarcity of comprehensive event datasets has limited performance potential in event modality. To address this challenge, we adapt CLIP`s architecture to align event embeddings with image embeddings, supporting zero-shot learning and preserving text alignment while mitigating catastrophic forgetting. Our encoder achieves strong performance in object recognition, with competitive results in zero-shot and few-shot learning tasks. Notably, it generalizes effectively to events extracted from video data without requiring additional training, highlighting its versatility. Additionally, we integrate this encoder within a cross-modality framework that facilitates interaction across five modalities-Image, Event, Text, Sound, and Depth-expanding the possibilities for cross-modal applications. Overall, this work underscores the transformative potential of a robust event encoder, broadening the scope and utility of event-based data across various fields. |
| title | Expanding Event Modality Applications through a Robust CLIP-Based Encoder |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.03093 |