UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908304252338176 |
|---|---|
| author | Jiao, Yang Qiu, Haibo Jie, Zequn Chen, Shaoxiang Chen, Jingjing Ma, Lin Jiang, Yu-Gang |
| author_facet | Jiao, Yang Qiu, Haibo Jie, Zequn Chen, Shaoxiang Chen, Jingjing Ma, Lin Jiang, Yu-Gang |
| contents | We introduce UniToken, an auto-regressive generation model that encodes visual inputs through a combination of discrete and continuous representations, enabling seamless integration of unified visual understanding and image generation tasks. Unlike previous approaches that rely on unilateral visual representations, our unified visual encoding framework captures both high-level semantics and low-level details, delivering multidimensional information that empowers heterogeneous tasks to selectively assimilate domain-specific knowledge based on their inherent characteristics. Through in-depth experiments, we uncover key principles for developing a unified model capable of both visual understanding and image generation. Extensive evaluations across a diverse range of prominent benchmarks demonstrate that UniToken achieves state-of-the-art performance, surpassing existing approaches. These results establish UniToken as a robust foundation for future research in this domain. The code and models are available at https://github.com/SxJyJay/UniToken. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04423 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding Jiao, Yang Qiu, Haibo Jie, Zequn Chen, Shaoxiang Chen, Jingjing Ma, Lin Jiang, Yu-Gang Computer Vision and Pattern Recognition Artificial Intelligence We introduce UniToken, an auto-regressive generation model that encodes visual inputs through a combination of discrete and continuous representations, enabling seamless integration of unified visual understanding and image generation tasks. Unlike previous approaches that rely on unilateral visual representations, our unified visual encoding framework captures both high-level semantics and low-level details, delivering multidimensional information that empowers heterogeneous tasks to selectively assimilate domain-specific knowledge based on their inherent characteristics. Through in-depth experiments, we uncover key principles for developing a unified model capable of both visual understanding and image generation. Extensive evaluations across a diverse range of prominent benchmarks demonstrate that UniToken achieves state-of-the-art performance, surpassing existing approaches. These results establish UniToken as a robust foundation for future research in this domain. The code and models are available at https://github.com/SxJyJay/UniToken. |
| title | UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.04423 |