Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917021468327936 |
|---|---|
| author | Yoshida, Naoki Hayakawa, Satoshi Takida, Yuhta Uesaka, Toshimitsu Wakaki, Hiromi Mitsufuji, Yuki |
| author_facet | Yoshida, Naoki Hayakawa, Satoshi Takida, Yuhta Uesaka, Toshimitsu Wakaki, Hiromi Mitsufuji, Yuki |
| contents | In this study, we propose an enhancement to the similarity computation mechanism in multi-modal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space. We theoretically prove that our method can approximate PMI with arbitrary accuracy and empirically demonstrate that our approach overall outperforms the standard CLIP formulation across several retrieval and classification tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15508 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity Yoshida, Naoki Hayakawa, Satoshi Takida, Yuhta Uesaka, Toshimitsu Wakaki, Hiromi Mitsufuji, Yuki Machine Learning Numerical Analysis In this study, we propose an enhancement to the similarity computation mechanism in multi-modal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space. We theoretically prove that our method can approximate PMI with arbitrary accuracy and empirically demonstrate that our approach overall outperforms the standard CLIP formulation across several retrieval and classification tasks. |
| title | Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2510.15508 |