LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915016466235392 |
|---|---|
| author | Wu, Wei Zheng, Kecheng Ma, Shuailei Lu, Fan Guo, Yuxin Zhang, Yifei Chen, Wei Guo, Qingpei Shen, Yujun Zha, Zheng-Jun |
| author_facet | Wu, Wei Zheng, Kecheng Ma, Shuailei Lu, Fan Guo, Yuxin Zhang, Yifei Chen, Wei Guo, Qingpei Shen, Yujun Zha, Zheng-Jun |
| contents | Understanding long text is of great demands in practice but beyond the reach of most language-image pre-training (LIP) models. In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method demonstrates superior performance in long-text-image retrieval tasks. The project page is available at https://wuw2019.github.io/lot-lip. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05249 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LoTLIP: Improving Language-Image Pre-training for Long Text Understanding Wu, Wei Zheng, Kecheng Ma, Shuailei Lu, Fan Guo, Yuxin Zhang, Yifei Chen, Wei Guo, Qingpei Shen, Yujun Zha, Zheng-Jun Computer Vision and Pattern Recognition Understanding long text is of great demands in practice but beyond the reach of most language-image pre-training (LIP) models. In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method demonstrates superior performance in long-text-image retrieval tasks. The project page is available at https://wuw2019.github.io/lot-lip. |
| title | LoTLIP: Improving Language-Image Pre-training for Long Text Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.05249 |