UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917027781804032 |
|---|---|
| author | An, Ruichuan Yang, Sihan Zhang, Renrui Shen, Zijun Lu, Ming Dai, Gaole Liang, Hao Guo, Ziyu Yan, Shilin Luo, Yulin Zou, Bocheng Yang, Chaoqun Zhang, Wentao |
| author_facet | An, Ruichuan Yang, Sihan Zhang, Renrui Shen, Zijun Lu, Ming Dai, Gaole Liang, Hao Guo, Ziyu Yan, Shilin Luo, Yulin Zou, Bocheng Yang, Chaoqun Zhang, Wentao |
| contents | Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation \textit{\textbf{personalized attribute-reasoning generation}}. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{https://github.com/arctanxarc/UniCTokens}{https://github.com/arctanxarc/UniCTokens}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14671 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens An, Ruichuan Yang, Sihan Zhang, Renrui Shen, Zijun Lu, Ming Dai, Gaole Liang, Hao Guo, Ziyu Yan, Shilin Luo, Yulin Zou, Bocheng Yang, Chaoqun Zhang, Wentao Computer Vision and Pattern Recognition Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation \textit{\textbf{personalized attribute-reasoning generation}}. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{https://github.com/arctanxarc/UniCTokens}{https://github.com/arctanxarc/UniCTokens}. |
| title | UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.14671 |