Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.10395 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908447046369280 |
|---|---|
| author | Xu, Zhiyang Chen, Jiuhai Lin, Zhaojiang Pan, Xichen Huang, Lifu Zhou, Tianyi Khabsa, Madian Wang, Qifan Jin, Di Yasunaga, Michihiro Yu, Lili Lin, Xi Victoria Nie, Shaoliang |
| author_facet | Xu, Zhiyang Chen, Jiuhai Lin, Zhaojiang Pan, Xichen Huang, Lifu Zhou, Tianyi Khabsa, Madian Wang, Qifan Jin, Di Yasunaga, Michihiro Yu, Lili Lin, Xi Victoria Nie, Shaoliang |
| contents | Recent advances in large language models (LLMs) have enabled multimodal foundation models to tackle both image understanding and generation within a unified framework. Despite these gains, unified models often underperform compared to specialized models in either task. A key challenge in developing unified models lies in the inherent differences between the visual features needed for image understanding versus generation, as well as the distinct training processes required for each modality. In this work, we introduce Pisces, an auto-regressive multimodal foundation model that addresses this challenge through a novel decoupled visual encoding architecture and tailored training techniques optimized for multimodal generation. Combined with meticulous data curation, pretraining, and finetuning, Pisces achieves competitive performance in both image understanding and image generation. We evaluate Pisces on over 20 public benchmarks for image understanding, where it demonstrates strong performance across a wide range of tasks. Additionally, on GenEval, a widely adopted benchmark for image generation, Pisces exhibits robust generative capabilities. Our extensive analysis reveals the synergistic relationship between image understanding and generation, and the benefits of using separate visual encoders, advancing the field of unified multimodal models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10395 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation Xu, Zhiyang Chen, Jiuhai Lin, Zhaojiang Pan, Xichen Huang, Lifu Zhou, Tianyi Khabsa, Madian Wang, Qifan Jin, Di Yasunaga, Michihiro Yu, Lili Lin, Xi Victoria Nie, Shaoliang Computer Vision and Pattern Recognition Artificial Intelligence Recent advances in large language models (LLMs) have enabled multimodal foundation models to tackle both image understanding and generation within a unified framework. Despite these gains, unified models often underperform compared to specialized models in either task. A key challenge in developing unified models lies in the inherent differences between the visual features needed for image understanding versus generation, as well as the distinct training processes required for each modality. In this work, we introduce Pisces, an auto-regressive multimodal foundation model that addresses this challenge through a novel decoupled visual encoding architecture and tailored training techniques optimized for multimodal generation. Combined with meticulous data curation, pretraining, and finetuning, Pisces achieves competitive performance in both image understanding and image generation. We evaluate Pisces on over 20 public benchmarks for image understanding, where it demonstrates strong performance across a wide range of tasks. Additionally, on GenEval, a widely adopted benchmark for image generation, Pisces exhibits robust generative capabilities. Our extensive analysis reveals the synergistic relationship between image understanding and generation, and the benefits of using separate visual encoders, advancing the field of unified multimodal models. |
| title | Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2506.10395 |