SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915177849421824 |
|---|---|
| author | Ge, Yuying Zhao, Sijie Zhu, Jinguo Ge, Yixiao Yi, Kun Song, Lin Li, Chen Ding, Xiaohan Shan, Ying |
| author_facet | Ge, Yuying Zhao, Sijie Zhu, Jinguo Ge, Yixiao Yi, Kun Song, Lin Li, Chen Ding, Xiaohan Shan, Ying |
| contents | The rapid evolution of multimodal foundation model has demonstrated significant progresses in vision-language understanding and generation, e.g., our previous work SEED-LLaMA. However, there remains a gap between its capability and the real-world applicability, primarily due to the model's limited capacity to effectively respond to various user instructions and interact with diverse visual data. In this work, we focus on bridging this gap through integrating two enhanced features: (1) comprehending images of arbitrary sizes and ratios, and (2) enabling multi-granularity image generation. We present a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks. Besides the competitive results on public benchmarks, SEED-X demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning. We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications. The models, codes, and datasets are released in https://github.com/AILab-CVC/SEED-X. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14396 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation Ge, Yuying Zhao, Sijie Zhu, Jinguo Ge, Yixiao Yi, Kun Song, Lin Li, Chen Ding, Xiaohan Shan, Ying Computer Vision and Pattern Recognition The rapid evolution of multimodal foundation model has demonstrated significant progresses in vision-language understanding and generation, e.g., our previous work SEED-LLaMA. However, there remains a gap between its capability and the real-world applicability, primarily due to the model's limited capacity to effectively respond to various user instructions and interact with diverse visual data. In this work, we focus on bridging this gap through integrating two enhanced features: (1) comprehending images of arbitrary sizes and ratios, and (2) enabling multi-granularity image generation. We present a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks. Besides the competitive results on public benchmarks, SEED-X demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning. We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications. The models, codes, and datasets are released in https://github.com/AILab-CVC/SEED-X. |
| title | SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.14396 |