ITO: Images and Texts as One via Synergizing Multiple Alignment and Training-Time Fusion
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912953821822976 |
|---|---|
| author | Liu, Hanpeng Li, Yaqian Wang, Zidan Zhang, Shuoxi Zhao, Zonglin Bo, Zihao Takezoe, Rinyoichi Long, Kaiwen He, Kun |
| author_facet | Liu, Hanpeng Li, Yaqian Wang, Zidan Zhang, Shuoxi Zhao, Zonglin Bo, Zihao Takezoe, Rinyoichi Long, Kaiwen He, Kun |
| contents | Image-text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield representations that remain partially organized by modality. We propose ITO, a framework addressing this limitation through two synergistic mechanisms. Multimodal multiple alignment enriches supervision by mining diverse image-text correspondences, while a lightweight training-time multimodal fusion module enforces structured cross-modal interaction. Crucially, the fusion module is discarded at inference, preserving the efficiency of standard dual-encoder architectures. Extensive experiments show that ITO consistently outperforms strong baselines across classification, retrieval, and multimodal benchmarks. Our analysis reveals that while multiple alignment drives discriminative power, training-time fusion acts as a critical structural regularizer -- eliminating the modality gap and stabilizing training dynamics to prevent the early saturation often observed in aggressive contrastive learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02767 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ITO: Images and Texts as One via Synergizing Multiple Alignment and Training-Time Fusion Liu, Hanpeng Li, Yaqian Wang, Zidan Zhang, Shuoxi Zhao, Zonglin Bo, Zihao Takezoe, Rinyoichi Long, Kaiwen He, Kun Computer Vision and Pattern Recognition Artificial Intelligence Image-text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield representations that remain partially organized by modality. We propose ITO, a framework addressing this limitation through two synergistic mechanisms. Multimodal multiple alignment enriches supervision by mining diverse image-text correspondences, while a lightweight training-time multimodal fusion module enforces structured cross-modal interaction. Crucially, the fusion module is discarded at inference, preserving the efficiency of standard dual-encoder architectures. Extensive experiments show that ITO consistently outperforms strong baselines across classification, retrieval, and multimodal benchmarks. Our analysis reveals that while multiple alignment drives discriminative power, training-time fusion acts as a critical structural regularizer -- eliminating the modality gap and stabilizing training dynamics to prevent the early saturation often observed in aggressive contrastive learning. |
| title | ITO: Images and Texts as One via Synergizing Multiple Alignment and Training-Time Fusion |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2603.02767 |