DreamLLM: Synergistic Multimodal Comprehension and Creation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929278564696064 |
|---|---|
| author | Dong, Runpei Han, Chunrui Peng, Yuang Qi, Zekun Ge, Zheng Yang, Jinrong Zhao, Liang Sun, Jianjian Zhou, Hongyu Wei, Haoran Kong, Xiangwen Zhang, Xiangyu Ma, Kaisheng Yi, Li |
| author_facet | Dong, Runpei Han, Chunrui Peng, Yuang Qi, Zekun Ge, Zheng Yang, Jinrong Zhao, Liang Sun, Jianjian Zhou, Hongyu Wei, Haoran Kong, Xiangwen Zhang, Xiangyu Ma, Kaisheng Yi, Li |
| contents | This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_11499 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DreamLLM: Synergistic Multimodal Comprehension and Creation Dong, Runpei Han, Chunrui Peng, Yuang Qi, Zekun Ge, Zheng Yang, Jinrong Zhao, Liang Sun, Jianjian Zhou, Hongyu Wei, Haoran Kong, Xiangwen Zhang, Xiangyu Ma, Kaisheng Yi, Li Computer Vision and Pattern Recognition Computation and Language Machine Learning This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io. |
| title | DreamLLM: Synergistic Multimodal Comprehension and Creation |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2309.11499 |