Auto-Encoding Morph-Tokens for Multimodal LLM
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911864845238272 |
|---|---|
| author | Pan, Kaihang Tang, Siliang Li, Juncheng Fan, Zhaoyu Chow, Wei Yan, Shuicheng Chua, Tat-Seng Zhuang, Yueting Zhang, Hanwang |
| author_facet | Pan, Kaihang Tang, Siliang Li, Juncheng Fan, Zhaoyu Chow, Wei Yan, Shuicheng Chua, Tat-Seng Zhuang, Yueting Zhang, Hanwang |
| contents | For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge. This is due to a conflicting objective: for comprehension, an MLLM needs to abstract the visuals; for generation, it needs to preserve the visuals as much as possible. Thus, the objective is a dilemma for visual-tokens. To resolve the conflict, we propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts; for generation, they take on a different, non-conflicting role as complete visual-tokens for image reconstruction, where the missing visual cues are recovered by the MLLM. Extensive experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously. Our project is available at https://github.com/DCDmllm/MorphTokens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01926 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Auto-Encoding Morph-Tokens for Multimodal LLM Pan, Kaihang Tang, Siliang Li, Juncheng Fan, Zhaoyu Chow, Wei Yan, Shuicheng Chua, Tat-Seng Zhuang, Yueting Zhang, Hanwang Computer Vision and Pattern Recognition For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge. This is due to a conflicting objective: for comprehension, an MLLM needs to abstract the visuals; for generation, it needs to preserve the visuals as much as possible. Thus, the objective is a dilemma for visual-tokens. To resolve the conflict, we propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts; for generation, they take on a different, non-conflicting role as complete visual-tokens for image reconstruction, where the missing visual cues are recovered by the MLLM. Extensive experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously. Our project is available at https://github.com/DCDmllm/MorphTokens. |
| title | Auto-Encoding Morph-Tokens for Multimodal LLM |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.01926 |