Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.14206 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916427095605248 |
|---|---|
| author | Liang, Guotao Zhang, Baoquan Wang, Yaowei Li, Xutao Ye, Yunming Wang, Huaibin Luo, Chuyao Ye, Kola Luo, linfeng |
| author_facet | Liang, Guotao Zhang, Baoquan Wang, Yaowei Li, Xutao Ye, Yunming Wang, Huaibin Luo, Chuyao Ye, Kola Luo, linfeng |
| contents | Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims to learn a codebook to encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. Although existing methods have shown superior performance, most methods prefer to learn a single-modal codebook (\emph{e.g.}, image), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (\emph{e.g.}, text-to-image, image captioning) due to the existence of modal gaps. In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to learn a codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre-trained text semantics as prior knowledge, then design two novel alignment modules (\emph{i.e.}, Semantic Alignment Module, and Relationship Alignment Module) to transfer such prior knowledge into codes for achieving codebook text alignment. In particular, our LG-VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that our method achieves superior performance on reconstruction and various multi-modal downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14206 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LG-VQ: Language-Guided Codebook Learning Liang, Guotao Zhang, Baoquan Wang, Yaowei Li, Xutao Ye, Yunming Wang, Huaibin Luo, Chuyao Ye, Kola Luo, linfeng Computer Vision and Pattern Recognition Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims to learn a codebook to encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. Although existing methods have shown superior performance, most methods prefer to learn a single-modal codebook (\emph{e.g.}, image), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (\emph{e.g.}, text-to-image, image captioning) due to the existence of modal gaps. In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to learn a codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre-trained text semantics as prior knowledge, then design two novel alignment modules (\emph{i.e.}, Semantic Alignment Module, and Relationship Alignment Module) to transfer such prior knowledge into codes for achieving codebook text alignment. In particular, our LG-VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that our method achieves superior performance on reconstruction and various multi-modal downstream tasks. |
| title | LG-VQ: Language-Guided Codebook Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.14206 |