Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910378250731520 |
|---|---|
| author | Jin, Yang Xu, Kun Xu, Kun Chen, Liwei Liao, Chao Tan, Jianchao Huang, Quzhe Chen, Bin Lei, Chenyi Liu, An Song, Chengru Lei, Xiaoqiang Zhang, Di Ou, Wenwu Gai, Kun Mu, Yadong |
| author_facet | Jin, Yang Xu, Kun Xu, Kun Chen, Liwei Liao, Chao Tan, Jianchao Huang, Quzhe Chen, Bin Lei, Chenyi Liu, An Song, Chengru Lei, Xiaoqiang Zhang, Di Ou, Wenwu Gai, Kun Mu, Yadong |
| contents | Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at https://github.com/jy0205/LaVIT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_04669 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization Jin, Yang Xu, Kun Xu, Kun Chen, Liwei Liao, Chao Tan, Jianchao Huang, Quzhe Chen, Bin Lei, Chenyi Liu, An Song, Chengru Lei, Xiaoqiang Zhang, Di Ou, Wenwu Gai, Kun Mu, Yadong Computer Vision and Pattern Recognition Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at https://github.com/jy0205/LaVIT. |
| title | Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2309.04669 |