t-gems: text-guided exit modules for decreasing clip image encoder
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913136999661568 |
|---|---|
| author | Presta, Alberto Stefanski, Grzegorz Byra, Michal Arendt, Krzysztof |
| author_facet | Presta, Alberto Stefanski, Grzegorz Byra, Michal Arendt, Krzysztof |
| contents | Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_17499 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | t-gems: text-guided exit modules for decreasing clip image encoder Presta, Alberto Stefanski, Grzegorz Byra, Michal Arendt, Krzysztof Machine Learning Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance. |
| title | t-gems: text-guided exit modules for decreasing clip image encoder |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.17499 |