Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.08949 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914799386886144 |
|---|---|
| author | Zhao, Xiangyu Liu, Bo Liu, Qijiong Shi, Guangyuan Wu, Xiao-Ming |
| author_facet | Zhao, Xiangyu Liu, Bo Liu, Qijiong Shi, Guangyuan Wu, Xiao-Ming |
| contents | We present EasyGen, an efficient model designed to enhance multimodal understanding and generation by harnessing the capabilities of diffusion models and large language models (LLMs), Unlike existing multimodal models that predominately depend on encoders like CLIP or ImageBind and need ample amounts of training data to bridge modalities,EasyGen leverages BiDiffuser,a bidirectional conditional diffusion model, to foster more efficient modality interactions. Easygen achieves text generation by training a projection layer linking BiDiffuser and an LLM, and facilities image generation by training an adapter to align the LLM's text space with the BiDiffuser's image space, Comprehensive quantitative and qualitative experiments show that EasyGen excels in data-efficient training, high-quality image generation, and extendibility, effectively addressing the challenges in multimodal generation. The source code is available at https://github.com/zxy556677/EasyGen. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_08949 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs Zhao, Xiangyu Liu, Bo Liu, Qijiong Shi, Guangyuan Wu, Xiao-Ming Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition We present EasyGen, an efficient model designed to enhance multimodal understanding and generation by harnessing the capabilities of diffusion models and large language models (LLMs), Unlike existing multimodal models that predominately depend on encoders like CLIP or ImageBind and need ample amounts of training data to bridge modalities,EasyGen leverages BiDiffuser,a bidirectional conditional diffusion model, to foster more efficient modality interactions. Easygen achieves text generation by training a projection layer linking BiDiffuser and an LLM, and facilities image generation by training an adapter to align the LLM's text space with the BiDiffuser's image space, Comprehensive quantitative and qualitative experiments show that EasyGen excels in data-efficient training, high-quality image generation, and extendibility, effectively addressing the challenges in multimodal generation. The source code is available at https://github.com/zxy556677/EasyGen. |
| title | EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2310.08949 |