Towards More Unified In-context Visual Understanding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909138837045248 |
|---|---|
| author | Sheng, Dianmo Chen, Dongdong Tan, Zhentao Liu, Qiankun Chu, Qi Bao, Jianmin Gong, Tao Liu, Bin Xu, Shengwei Yu, Nenghai |
| author_facet | Sheng, Dianmo Chen, Dongdong Tan, Zhentao Liu, Qiankun Chu, Qi Bao, Jianmin Gong, Tao Liu, Bin Xu, Shengwei Yu, Nenghai |
| contents | The rapid advancement of large language models (LLMs) has accelerated the emergence of in-context learning (ICL) as a cutting-edge approach in the natural language processing domain. Recently, ICL has been employed in visual understanding tasks, such as semantic segmentation and image captioning, yielding promising results. However, existing visual ICL framework can not enable producing content across multiple modalities, which limits their potential usage scenarios. To address this issue, we present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space, structured as interleaved in-context sequences. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them, facilitating in-context learning. Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline.Experimental results demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines. Overall, our research takes a further step toward unified multimodal in-context learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_02520 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Towards More Unified In-context Visual Understanding Sheng, Dianmo Chen, Dongdong Tan, Zhentao Liu, Qiankun Chu, Qi Bao, Jianmin Gong, Tao Liu, Bin Xu, Shengwei Yu, Nenghai Computer Vision and Pattern Recognition The rapid advancement of large language models (LLMs) has accelerated the emergence of in-context learning (ICL) as a cutting-edge approach in the natural language processing domain. Recently, ICL has been employed in visual understanding tasks, such as semantic segmentation and image captioning, yielding promising results. However, existing visual ICL framework can not enable producing content across multiple modalities, which limits their potential usage scenarios. To address this issue, we present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space, structured as interleaved in-context sequences. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them, facilitating in-context learning. Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline.Experimental results demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines. Overall, our research takes a further step toward unified multimodal in-context learning. |
| title | Towards More Unified In-context Visual Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.02520 |