COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909119226576896 |
|---|---|
| author | Jiang, Chaoya Xu, Haiyang Ye, Wei Ye, Qinghao Li, Chenliang Yan, Ming Bi, Bin Zhang, Shikun Zhang, Ji Huang, Fei |
| author_facet | Jiang, Chaoya Xu, Haiyang Ye, Wei Ye, Qinghao Li, Chenliang Yan, Ming Bi, Bin Zhang, Shikun Zhang, Ji Huang, Fei |
| contents | Vision-Language Pre-training (VLP) methods based on object detection enjoy the rich knowledge of fine-grained object-text alignment but at the cost of computationally expensive inference. Recent Visual-Transformer (ViT)-based approaches circumvent this issue while struggling with long visual sequences without detailed cross-modal alignment information. This paper introduces a ViT-based VLP technique that efficiently incorporates object information through a novel patch-text alignment mechanism. Specifically, we convert object-level signals into patch-level ones and devise a Patch-Text Alignment pre-training task (PTA) to learn a text-aware patch detector. By using off-the-shelf delicate object annotations in 5\% training images, we jointly train PTA with other conventional VLP objectives in an end-to-end manner, bypassing the high computational cost of object detection and yielding an effective patch detector that accurately detects text-relevant patches, thus considerably reducing patch sequences and accelerating computation within the ViT backbone. Our experiments on a variety of widely-used benchmarks reveal that our method achieves a speedup of nearly 88\% compared to prior VLP models while maintaining competitive or superior performance on downstream tasks with similar model size and data scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_03475 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment Jiang, Chaoya Xu, Haiyang Ye, Wei Ye, Qinghao Li, Chenliang Yan, Ming Bi, Bin Zhang, Shikun Zhang, Ji Huang, Fei Multimedia Vision-Language Pre-training (VLP) methods based on object detection enjoy the rich knowledge of fine-grained object-text alignment but at the cost of computationally expensive inference. Recent Visual-Transformer (ViT)-based approaches circumvent this issue while struggling with long visual sequences without detailed cross-modal alignment information. This paper introduces a ViT-based VLP technique that efficiently incorporates object information through a novel patch-text alignment mechanism. Specifically, we convert object-level signals into patch-level ones and devise a Patch-Text Alignment pre-training task (PTA) to learn a text-aware patch detector. By using off-the-shelf delicate object annotations in 5\% training images, we jointly train PTA with other conventional VLP objectives in an end-to-end manner, bypassing the high computational cost of object detection and yielding an effective patch detector that accurately detects text-relevant patches, thus considerably reducing patch sequences and accelerating computation within the ViT backbone. Our experiments on a variety of widely-used benchmarks reveal that our method achieves a speedup of nearly 88\% compared to prior VLP models while maintaining competitive or superior performance on downstream tasks with similar model size and data scale. |
| title | COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment |
| topic | Multimedia |
| url | https://arxiv.org/abs/2308.03475 |