Improving fine-grained understanding in image-text pre-training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911760261316608 |
|---|---|
| author | Bica, Ioana Ilić, Anastasija Bauer, Matthias Erdogan, Goker Bošnjak, Matko Kaplanis, Christos Gritsenko, Alexey A. Minderer, Matthias Blundell, Charles Pascanu, Razvan Mitrović, Jovana |
| author_facet | Bica, Ioana Ilić, Anastasija Bauer, Matthias Erdogan, Goker Bošnjak, Matko Kaplanis, Christos Gritsenko, Alexey A. Minderer, Matthias Blundell, Charles Pascanu, Razvan Mitrović, Jovana |
| contents | We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09865 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving fine-grained understanding in image-text pre-training Bica, Ioana Ilić, Anastasija Bauer, Matthias Erdogan, Goker Bošnjak, Matko Kaplanis, Christos Gritsenko, Alexey A. Minderer, Matthias Blundell, Charles Pascanu, Razvan Mitrović, Jovana Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models. |
| title | Improving fine-grained understanding in image-text pre-training |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2401.09865 |