HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915608649531392 |
|---|---|
| author | Wu, Ruijia Chen, Ping Shen, Fei Zhao, Shaoan Hui, Qiang Gao, Huanlin Lu, Ting Liu, Zhaoxiang Zhao, Fang Wang, Kai Lian, Shiguo |
| author_facet | Wu, Ruijia Chen, Ping Shen, Fei Zhao, Shaoan Hui, Qiang Gao, Huanlin Lu, Ting Liu, Zhaoxiang Zhao, Fang Wang, Kai Lian, Shiguo |
| contents | Contrastive vision-language models like CLIP have achieved impressive results in image-text retrieval by aligning image and text representations in a shared embedding space. However, these models often treat text as flat sequences, limiting their ability to handle complex, compositional, and long-form descriptions. In particular, they fail to capture two essential properties of language: semantic hierarchy, which reflects the multi-level compositional structure of text, and semantic monotonicity, where richer descriptions should result in stronger alignment with visual content.To address these limitations, we propose HiMo-CLIP, a representation-level framework that enhances CLIP-style models without modifying the encoder architecture. HiMo-CLIP introduces two key components: a hierarchical decomposition (HiDe) module that extracts latent semantic components from long-form text via in-batch PCA, enabling flexible, batch-aware alignment across different semantic granularities, and a monotonicity-aware contrastive loss (MoLo) that jointly aligns global and component-level representations, encouraging the model to internalize semantic ordering and alignment strength as a function of textual completeness.These components work in concert to produce structured, cognitively-aligned cross-modal representations. Experiments on multiple image-text retrieval benchmarks show that HiMo-CLIP consistently outperforms strong baselines, particularly under long or compositional descriptions. The code is available at https://github.com/UnicomAI/HiMo-CLIP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_06653 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment Wu, Ruijia Chen, Ping Shen, Fei Zhao, Shaoan Hui, Qiang Gao, Huanlin Lu, Ting Liu, Zhaoxiang Zhao, Fang Wang, Kai Lian, Shiguo Computer Vision and Pattern Recognition Computation and Language Contrastive vision-language models like CLIP have achieved impressive results in image-text retrieval by aligning image and text representations in a shared embedding space. However, these models often treat text as flat sequences, limiting their ability to handle complex, compositional, and long-form descriptions. In particular, they fail to capture two essential properties of language: semantic hierarchy, which reflects the multi-level compositional structure of text, and semantic monotonicity, where richer descriptions should result in stronger alignment with visual content.To address these limitations, we propose HiMo-CLIP, a representation-level framework that enhances CLIP-style models without modifying the encoder architecture. HiMo-CLIP introduces two key components: a hierarchical decomposition (HiDe) module that extracts latent semantic components from long-form text via in-batch PCA, enabling flexible, batch-aware alignment across different semantic granularities, and a monotonicity-aware contrastive loss (MoLo) that jointly aligns global and component-level representations, encouraging the model to internalize semantic ordering and alignment strength as a function of textual completeness.These components work in concert to produce structured, cognitively-aligned cross-modal representations. Experiments on multiple image-text retrieval benchmarks show that HiMo-CLIP consistently outperforms strong baselines, particularly under long or compositional descriptions. The code is available at https://github.com/UnicomAI/HiMo-CLIP. |
| title | HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2511.06653 |