TULIP: Token-length Upgraded CLIP
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909556136738816 |
|---|---|
| author | Najdenkoska, Ivona Derakhshani, Mohammad Mahdi Asano, Yuki M. van Noord, Nanne Worring, Marcel Snoek, Cees G. M. |
| author_facet | Najdenkoska, Ivona Derakhshani, Mohammad Mahdi Asano, Yuki M. van Noord, Nanne Worring, Marcel Snoek, Cees G. M. |
| contents | We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10034 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TULIP: Token-length Upgraded CLIP Najdenkoska, Ivona Derakhshani, Mohammad Mahdi Asano, Yuki M. van Noord, Nanne Worring, Marcel Snoek, Cees G. M. Computer Vision and Pattern Recognition We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip. |
| title | TULIP: Token-length Upgraded CLIP |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.10034 |